Major changes:
Jobs are not pulled by worker, they are distributed by the library to the specific celery queue defined by the jobs arch/num_cu. The workers in the queue each get assigned 1 job and the config for the job. They run, and write back to mySQL.
Jobs that execute through celery: fin_find_compile and fin_find_eval.
All other tuna commands run as they did before.
The docker-compose launches the celery docker, the cmd executing in the docker is the celery worker command that listens for jobs: celery -A tuna.celery_app.celery worker -l info -E
The jobs are launched separately in a shell by the go_fish command.
Draft to get some eyes on code changes.
Major changes: Jobs are not pulled by worker, they are distributed by the library to the specific celery queue defined by the jobs arch/num_cu. The workers in the queue each get assigned 1 job and the config for the job. They run, and write back to mySQL.
Jobs that execute through celery: fin_find_compile and fin_find_eval. All other tuna commands run as they did before.
The docker-compose launches the celery docker, the cmd executing in the docker is the celery worker command that listens for jobs:
celery -A tuna.celery_app.celery worker -l info -EThe jobs are launched separately in a shell by the go_fish command.Sample run: shell 1:
shell 2 (source correct env for tuning)