Operator changes to add a new executor pod that is responsible for handling background tasks that are currently handled in wandb-app.
With this change, wandb-app and parquet just schedule tasks using async, which is backed by the existing Redis instance. Then the executor pod is responsible for actually kicking this off. This has a few advantages:
If pods restart, because the tasks are still in redis, the pods will pick the work back up.
We offload heavy work off of wandb-app which we know causes instability for customers.
We could autoscale executor to better scale with specific customer workloads.
Operator changes to add a new
executorpod that is responsible for handling background tasks that are currently handled inwandb-app.With this change,
wandb-appandparquetjust schedule tasks using async, which is backed by the existing Redis instance. Then theexecutorpod is responsible for actually kicking this off. This has a few advantages:wandb-appwhich we know causes instability for customers.