pratacosmin
commented
1 month ago I took a look at the https://etcd.io/ project and for me look a little bit complicated maybe will be easier to use something like RabbitMq? Anyway this change will require us to have a producer that will read unique traces from jaeger and put them to the queue/database, a consumer and also the service for queen or db.
This part will require some work but the gain will be processing power for traces using multiple consumers.
Right now Pythia is only tested in standalone mode, we need to ensure that Pythia can work as as a service in a k8s cluster and can be scaled to handle the volume of traces in production.
This means that the multiple Pythia containers might be running at the same time and we need to ensure that they don't step on each other and end up processing the same traces
Which means the pythia containers need to figure out how to co-ordinate amongst each other maybe using a distributed key value store like https://etcd.io/ ?