DataCater / datacater

The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
https://datacater.io
Other
82 stars 4 forks source link

Locking Python Runner Retrieval #27

Closed HknLof closed 1 year ago

HknLof commented 1 year ago

Right now the python runner retrieval works asynchronously. This could potentially lead to using a single runner multiple times. To get rid of this problem two approaches are possible.

  1. Use vertx cluster wide locks and retrieve the AsyncMap only while having an active lock.
  2. Mark created Kubernetes pods as being in use before delivering payload.

Upon start implementation decide for option one or two and the reasoning behind it.

Refs: This issue is a follow up to