stripe / veneur

A distributed, fault-tolerant pipeline for observability data
MIT License
1.74k stars 174 forks source link

Stripe/veneur docker images fail using default configuration with obtuse error #970

Open ramfjord opened 2 years ago

ramfjord commented 2 years ago

I've attempted to use the official seeming images tag versions 14.2.0 (then noticed the releases section doesn't say this is in released status), 14.1.0, 13.2.0 so far. These are failing with an error like this:

$ docker run -t stripe/veneur:14.1.0 ./veneur -f config.yaml
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
    A future release of golang/protobuf will delete this package,
    which has long been excluded from the compatibility promise.

INFO[0000] Set mutex profile fraction                    MutexProfileFraction=0 previousMutexProfileFraction=0
INFO[0000] Set block profile rate (nanoseconds)          BlockProfileRate=0
INFO[0000] Preparing workers                             number=96
INFO[0000] Creating X-Ray client                         Address="localhost:2000"
INFO[0000] Configured X-Ray span sink                    num_annotation_tags="[]" sample_percentage=100
INFO[0000] Configured Falconer trace sink
INFO[0000] Configured Prometheus metric sink.
INFO[0000] Creating SignalFx sink signalfx               endpoint_base="https://ingest.signalfx.com" metric_sink=signalfx
WARN[0000] Unknown sink kind signalfx; skipping.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x118d5fc]

goroutine 1 [running]:
github.com/stripe/veneur/v14.(*Server).createSpanSinks(0x166b140, 0xc000208d20, 0xc00041d278, 0xc000592730)
    /veneur/server.go:324 +0x1bc
github.com/stripe/veneur/v14.NewFromConfig({{{0xc00044ced0, 0x3, 0x3}, {{0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, {{0x0, ...}}, ...}, ...})
    /veneur/server.go:716 +0x2025
main.main()
    /veneur/cmd/veneur/main.go:75 +0x14fc

I was assuming this would have been built using the 14.1.0 release tag in the example above, but when I look at server.go:324 on that release tag, it's an empty line, which seems to me to prove that this binary was built... I have no idea where, but not using the 14.1.0 release tag! I'm aware this probably should fail in some way without supplying my datadog key, but it would be much easier for me to figure out what is wrong with my supplied configuration if there were at least some kind of error message related to what's wrong with the config here. I'm thinking my next step here should be to compile from source so I can actually know where my errors are coming from at least, but is this an officially maintained image that I'm trying to use? Also, is the current release 14.2.0 or 14.1.0? The release notes for 14.2.0 don't have a release date like the others.

ramfjord commented 2 years ago

Well, the specific issue on 14.1.0's default config is something I've fixed by going through all of the various sink config options (grepping for SpanSinks in code), and I have currently got my instance working by setting a bunch of options to empty string by default. The last one that got it finally working was disabling the various splunk options referenced here (still using stripe/veneur:14.1.0 image)

ramfjord commented 2 years ago

I suppose I'm asking for a few things here, and feel free to work on or deny any of them:

ramfjord commented 2 years ago

BTW it appears the 14.1.0 tag should probably be referencing the b73d1860 commit, which I found by checking the update time of the docker image, and seems to correspond with file/line numbers to the various error messages I was getting