Currently the sync app doesn't work unless concurrency is 1 due to no inter-state checking. Atomic tasks would mean only one task of each state executes.
The VeRSI-developed MyTardis sync app required Celery message queue concurrency to be set to 1 on deployments. This was because there were no provisions developed to ensure specific state tasks in the finite state machine were only run once. This was accepted in MyTardis 2.5 and deployed with a view to "fix it later".
MyTardis 3.0 takes advantage of the task system with many concurrent processes including file fetching, verification and ATOM harvesting. Concurrency = 1, particularly for long sync-app processes such as file existence checks would create massive inefficiencies for MyTardis servers.
Getting celery to run atomic tasks, Kieran speculates is a fairly trivial change. This change can be implemented outside of the synchrotron but would require testing with the synchrotron workflow.
Currently the sync app doesn't work unless concurrency is 1 due to no inter-state checking. Atomic tasks would mean only one task of each state executes.
The VeRSI-developed MyTardis sync app required Celery message queue concurrency to be set to 1 on deployments. This was because there were no provisions developed to ensure specific state tasks in the finite state machine were only run once. This was accepted in MyTardis 2.5 and deployed with a view to "fix it later".
MyTardis 3.0 takes advantage of the task system with many concurrent processes including file fetching, verification and ATOM harvesting. Concurrency = 1, particularly for long sync-app processes such as file existence checks would create massive inefficiencies for MyTardis servers.
Getting celery to run atomic tasks, Kieran speculates is a fairly trivial change. This change can be implemented outside of the synchrotron but would require testing with the synchrotron workflow.
original LH ticket
This ticket has 0 attachment(s).