astronomer / astro-provider-ray

This provider contains operators, decorators and triggers to send a ray job from an airflow task
https://astronomer.github.io/astro-provider-ray/
Apache License 2.0
12 stars 2 forks source link

Bugfix: Better exception handling and cluster clean up #68

Closed venkatajagannath closed 1 month ago

venkatajagannath commented 1 month ago

This PR fixes a bug for cleaning up cluster resources when they are no longer required.

We make 2 changes --

  1. SubmitRayJob -- Add better exception handling to ensure cluster is deleted if there is an exception in the execute() method
  2. RayJobTrigger -- Added cleanup methods to ensure that even if there is an exception when the trigger is running, the cluster clean up still happens. This was not the case earlier

Also added additional unit tests for both operator & trigger to ensure high code coverage.

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.44%. Comparing base (48b9785) to head (27a2020). Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
ray_provider/operators/ray.py 95.65% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #68 +/- ## ========================================== + Coverage 95.42% 96.44% +1.02% ========================================== Files 5 5 Lines 546 563 +17 ========================================== + Hits 521 543 +22 + Misses 25 20 -5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.