Libhoney events sent to honeycomb API return "x509: certificate signed by unknown authority" error.

[william-coulter]

MacOS Version: High Sierra 10.13.6 Golang Version: go1.12.6 darwin/amd64 Libhoney Version: v1.12.4 Client Docker Engine:

• Version: 19.03.2
• API version: 1.40
• Go version: go1.12.8
• OS/Arch: darwin/amd64
• Experimental: false

I'm trying to send an event to my dataset using the libhoney-go package. I'm following the first example at: https://docs.honeycomb.io/getting-data-in/go/sdk/#examples.

Here is a snippet of my code which is running in a local docker container

func main() {
    libhoney.Init(libhoney.Config{
        WriteKey: "KEY",
        Dataset: "DATASET",

    })

    defer libhoney.Close() // Flush any pending calls to Honeycomb

    ev := libhoney.NewEvent()
    ev.Add(map[string]interface{}{
        "duration_ms": 153.12,
        "method": "get",
        "hostname": "appserver15",
        "payload_length": 27,
    })

    log.Println("Sending event...")
    log.Println(ev)

    ev.Send()
    c := libhoney.Responses()

    log.Println("Waiting to receive response from honeycomb...")
    res := <-c
    log.Println(res)
}

The response returned from the libhoney.Responses() channel is:

x509: certificate signed by unknown authority

A lot of the docs online point to this being an issue with MacOS and Docker (https://forums.docker.com/t/docker-private-registry-x509-certificate-signed-by-unknown-authority/21262) but I've played around with my keychains and can't overcome this issue. I suspect keychains aren't the issue because I'm running the nodejs beeline package in a different docker container and it has no issues sending messages to Honeycomb.

I was wondering if something like this has come up previously with a different user? Any help would be appreciated, thanks :)

[lizthegrey]

What is the base image for the docker container you are running your application in? We use AWS's CA.

[william-coulter]

Hi Liz, I'm using Docker Desktop for Mac: https://hub.docker.com/editions/community/docker-ce-desktop-mac

[lizthegrey]

No, I need to see your Dockerfile, in particular the first line of it.

[william-coulter]

Ah yep, here you go.

We use stretch rather than buster because buster's libc is too high, meaning the binary can't be run on Ubuntu 18.04 LTS.

FROM golang:stretch AS builder
RUN apt-get update -y && apt-get install -y netcat
WORKDIR /app
...

FROM debian:stretch-slim
WORKDIR /app
COPY --from=builder /bin/nc /bin/
...

[lizthegrey]

I suspect Stretch is missing the AWS CA from its ca-certificates package. I suggest you install a more recent ca-certificates backport as part of your image generation process.

[william-coulter]

Yep so I've install the ca-certificates package with the following code snippet in my Dockerfile

# Add the stretch-backports repository to your package sources list
RUN sh -c 'echo "deb http://deb.debian.org/debian stretch-backports main" >> /etc/apt/sources.list.d/ca-certificates.list'
# Update the package cache and install netcat
RUN apt-get update -y && apt-get install -y netcat
# Install ca-certificates package from the stretch-backports repository
RUN apt-get -t stretch-backports install ca-certificates

Everything runs as usual but I'm still getting that x509: certificate signed by unknown authority error.

[lizthegrey]

https://packages.debian.org/search?keywords=ca-certificates&searchon=names&section=all&suite=stretch-backports

Nobody from Debian has backported it :(

[vreynolds]

Hello,

We will be closing this issue as it is a low priority for us. It is unlikely that we'll ever get to it, and so we'd like to set expectations accordingly.

As we enter 2022 Q1, we are trimming our OSS backlog. This is so that we can focus better on areas that are more aligned with the OpenTelemetry-focused direction of telemetry ingest for Honeycomb.

If this issue is important to you, please feel free to ping here and we can discuss/re-open.