One thing which should definititely be done is to modify the current health check in the API-server so that each time the background task runs it registers the dummy job_id and unregisters it again. The "leaking" of job_ids would probably be detected by this check.
Figure out if there is a bullet-proof way to ensure that a topic is deregistered. Would it e.g. be possible to ask the RabbitMq server periodically and check that all job_ids which are not streaming logs should be deregistered. I would like to discuss this with you @sanderegg
Probably this will also be a bottleneck if a sufficient number of users stream logs via the API server at the same time. So it might be worth considering ways to communicate this to the orchestrator to spin up additional replicas of the api-server.
Is there an existing issue for this?
Ideas and thoughts
job_id
and unregisters it again. The "leaking" ofjob_id
s would probably be detected by this check.job_id
s which are not streaming logs should be deregistered. I would like to discuss this with you @sanderegg