Introduce existing and new Celery scripts to the repository.
Why?
Without version control, tracking changes and discussing modifications to these scripts is challenging. Adding them to GitHub allows for better historical tracking and collaborative development.
How?
The integration includes pre-existing scripts used in the LSF environment. Celery workers are managed through celery_worker_scripts/<env>/START_CELERY_WORKERS.sh, which are refreshed every three days via crontab using celery_worker_scripts/<env>/start_celery_worker.sh. For the upcoming SLURM migration, new scripts ending in _SLURM.sh or _slurm.sh are added, alongside the implementation of scrontab to manage these tasks.
Testing?
In the development environment, the scripts are tested with shorter cron intervals (2-3 minutes) to ensure functionality. One can test validation as well since the NF executor name is set to slurm now.
For production, initiating scrontab before the data release won't impact the current LSF setup, allowing for a safe testing environment. Monitoring the health of celery workers post-implementation will be key to ensuring successful deployment.
What?
Introduce existing and new Celery scripts to the repository.
Why?
Without version control, tracking changes and discussing modifications to these scripts is challenging. Adding them to GitHub allows for better historical tracking and collaborative development.
How?
The integration includes pre-existing scripts used in the LSF environment. Celery workers are managed through
celery_worker_scripts/<env>/START_CELERY_WORKERS.sh, which are refreshed every three days via crontab usingcelery_worker_scripts/<env>/start_celery_worker.sh. For the upcoming SLURM migration, new scripts ending in_SLURM.shor_slurm.share added, alongside the implementation of scrontab to manage these tasks.Testing?
In the development environment, the scripts are tested with shorter cron intervals (2-3 minutes) to ensure functionality. One can test validation as well since the NF executor name is set to
slurmnow.For production, initiating scrontab before the data release won't impact the current LSF setup, allowing for a safe testing environment. Monitoring the health of celery workers post-implementation will be key to ensuring successful deployment.
Also see https://github.com/EBISPOT/goci/issues/1179