ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.45k stars 189 forks source link

Gracefully handle panics in UDFs #584

Closed mwylde closed 3 months ago

mwylde commented 3 months ago

If a UDF panics (across the current C FFI boundary) it causes the parent to abort. Currently, this leads to a double free and segfault, because both the UDF wrapper and the calling code attempt to free the input arrays. (In normal operation this doesn't happen because we were leaking the arrays from the parent; but the mem::forget call was after the invocation of the UDF and thus wasn't being called if that invocation panicked).

This PR fixes that issue by leaking the memory before we call the UDF. However, that still leads to an unwanted abort of the process; ideally we'd have our queue-driven clean shutdown like we have if a UDF panics in 0.9.

To get back to that, we now catch panics in the UDF wrapper code and return a bool indicating whether or not the UDF panicked. If so, we re-emit the panic in the parent.