search

clingen-data-model / clinvar-streams

1 stars 0 forks source link

Resume from snapshot #38

Closed theferrit32 closed 2 years ago

theferrit32 commented 2 years ago

This includes the combiner resume-from-snapshot functionality, and the graceful shutdown with PreStop http hook in both the combiner and raw code.

theferrit32 commented 2 years ago

This is ready to review.

clinvar-raw processing code has 3 separate threads, one for consuming, one for processing, and one for producing, so getting the graceful shutdown to work was a little more complicated. Ended up using a watcher thread that checks on those 3 threads, and when they're all terminated, flips a boolean atom.

https://github.com/clingen-data-model/clinvar-streams/pull/38/files#diff-61413bb2eb24a21b9aa0d69d7f3b02c9ade5a40b7d8f61601f156b186379a4c1R253-R258

Another way to have done this is to lean more heavily on the mount library.