open-telemetry / otel-arrow

Protocol and libraries for sending and receiving OpenTelemetry data using Apache Arrow
Apache License 2.0
83 stars 15 forks source link

More panic handling for receiver; print Error, not Debug #193

Closed jmacd closed 5 months ago

jmacd commented 5 months ago

This will help explain a crash we've seen internally -- it appears that unless Debug-level logging is enabled, we are not likely to see any indication when a panic happens inside the receiver. In v0.22.0 and earlier, this would be logged and recovered. With the changes in v0.23.0, these panics would interrupt the process because not all goroutines support panic recovery.

If the summary above holds, it means that the errors we've seen were not caused by v0.23.0 but were pre-existing conditions, and the only regression in v0.23.0 was not to recover from the panic.