Open jbeemster opened 4 years ago
Isolating this down it seems like something in this function is causing the issue: https://github.com/snowplow/snowplow-golang-tracker/blob/master/tracker/emitter.go#L100-L115
Going to see if the issue can be fixed by using a custom transport instead of the default one.
Also we are editing the global transport object in this way which feels like something we should be avoiding - it seems likely that providing a custom transport rather than using the default transport is almost certainly the way to clean this issue up and stop it from occurring.
Example of a default we could copy: https://github.com/golang/go/blob/go1.13.14/src/net/http/transport.go#L42-L54
The issue is line 109 which dereferences the pointer and therefore copies the mutex value.
I think it would be fixed by
defaultTransport := http.DefaultTransport.(*http.Transport).Clone()
A workaround is to pass a custom http client to InitEmitter().
Describe the bug When running the tracker there are occasional panics thrown which cause the whole application to crash. These occur not in the Tracker itself but in the Emitter transport layer below it - however it might be possible to handle this panic within the Emitter to recover gracefully.
To Reproduce This is not easy to reproduce but in a long running application I will see crashes consistently every 1-2 days.
Expected behavior The tracker should not throw a panic performing its core function.
Environment (please complete the following information):
alpine:3.7
Additional context The stack trace: