elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
71 stars 126 forks source link

Ensure orchestrator tasks are always canceled #2661

Closed seanstory closed 3 months ago

seanstory commented 3 months ago

relates to https://github.com/elastic/connectors/issues/2638

We've been having an issue with the GCS ftest taking > 10 minutes. Digging with @artem-shelkovnikov today, we found that there was a sync that received a SIGINT, and acted like it was stopping/cleaning up (printed job counters and everything) but the extractor clearly kept going. Digging further, we found a logical error, where the SyncOrchestrator's tasks were only being canceled if there was an "error" in the job.

This change instead makes sure that orchestrator.close() always cancels the tasks if they are still running. It also makes sure that the SyncJobRunner's _sync_done() function always calls orchestrator.cancel(). There is no risk in calling cancel() multiple times.

Checklists

Pre-Review Checklist

Release Note

Fixes a bug where the framework might receive an interrupt, but not terminate all running sync job tasks correctly.

github-actions[bot] commented 3 months ago

💚 Backport PR(s) successfully created

Status Branch Result
8.14 https://github.com/elastic/connectors/pull/2662

This backport PR will be merged automatically after passing CI.