search

lanl / BEE

Other
13 stars 3 forks source link

Processing queues even when there are no workflows #840

Open pagrubel opened 1 month ago

pagrubel commented 1 month ago

I put a debug statement to verify what happens when beeflow starts back up that caused me to realize that we constantly process our queues. We should probably only do that if there are any new workflows.

[2024-05-13 16:31:38,511] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:31:43,499] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:31:48,507] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:31:53,505] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:31:58,514] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:32:03,498] beeflow.task_manager.background:update_jobs(): queue count is 0 [2024-05-13 16:32:08,500] beeflow.task_manager.background:update_jobs():

jtronge commented 3 weeks ago

I'm wondering if we should switch this code to use celery for background jobs? I think the apscheduler code was originally a temporary solution and we're already using celery in the workflow manager.