Fix metadata logging even when actors crash

IBM / data-prep-kit

Open source project for data preparation of LLM application builders

https://ibm.github.io/data-prep-kit/

Apache License 2.0

316 stars 135 forks source link

Fix metadata logging even when actors crash #721

Closed shivdeep-singh-ibm closed 4 weeks ago

shivdeep-singh-ibm commented 1 month ago

Why are these changes needed?

In case the ray actors crash, the key processing_time may not be updated to stats, so while rounding off this key, we may get an exception which stops metadata from being written.

Related issue number (if any).

Also seen in the crash here. https://github.com/IBM/data-prep-kit/issues/719