open-telemetry / otel-arrow

Protocol and libraries for sending and receiving OpenTelemetry data using Apache Arrow
Apache License 2.0
68 stars 13 forks source link

More panic handling; print an Error, not Debug-level log #194

Closed jmacd closed 3 months ago

jmacd commented 3 months ago

This will help explain a crash we've seen internally -- it appears that unless Debug-level logging is enabled, we are not likely to see any indication when a panic happens inside the receiver. In v0.22.0 and earlier, this would be logged and recovered. With the changes in v0.23.0, these panics would interrupt the process because not all goroutines support panic recovery.

If the summary above holds, it means that the errors we've seen were not caused by v0.23.0 but were pre-existing conditions, and the only regression in v0.23.0 was not to recover from the panic.