clingen-data-model / clinvar-streams

1 stars 0 forks source link

Implement local stream offset management instead of using consumer groups. #66

Open theferrit32 opened 2 years ago

theferrit32 commented 2 years ago

Since the clinvar-raw producer application has a kubernetes Application-scoped persistent volume now, we can store additional info there like the latest processed offset (similar to how genegraph does it). When pod is redeployed or restarted or updated within the scope of the same Application, the persistent volume of the StatefulSet is not destroyed and the code can restart from the same offset stored there. If the StatefulSet is fully deleted and recreated, it will lose track of where it had processed to previously, but this is probably both okay and desirable.

theferrit32 commented 2 years ago

This will avoid any of the timeout issues related to this: https://github.com/clingen-data-model/clinvar-streams/issues/62