search

DataCater / datacater

The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
https://datacater.io
Other
82 stars 4 forks source link

Implementing pooling for python runner. #11

Closed HknLof closed 1 year ago

HknLof commented 2 years ago

To guarantee multi-tenancy we introduce pooling of our python runner. The idea is to have throw-away instances of the runner. After each request - response loop, these get discarded and automatically restarted by the underlying statefulset.

A non-functional requirement is to use a distributed data structure to ensure we can still horizontally scale.

HknLof commented 1 year ago

Left are some tests. Otherwise done. ETA Nov 07th

flippingbits commented 1 year ago

I believe we can close this issue. @HknLof Please reopen if I am wrong.