[ ] Decouple the workflow processing from each step (hashing, copying missing files, generating) such that each step may happen for the granule as soon as it is ready to happen. No need to wait for all granules to finish hashing before coping over missing files, etc.
[ ] Submit the Globus transfer from the pull_process.py script. Then, pass on the globus submit IDs to globus_push.py for it to wait for transfers to finish. This means that Globus transfers can happen while globus_push.py is still in the queue, thus saving time.
[ ] Provide better logging constructs by extending the use of the summary logs. Currently, summary logs are not properly logged for all parts of the code.
[ ] Determine and code in a better communication pattern between master and slave ranks. Maybe include this in a future BasicFusion Python package?
[ ] Decouple the workflow processing from each step (hashing, copying missing files, generating) such that each step may happen for the granule as soon as it is ready to happen. No need to wait for all granules to finish hashing before coping over missing files, etc.
[ ] Submit the Globus transfer from the pull_process.py script. Then, pass on the globus submit IDs to globus_push.py for it to wait for transfers to finish. This means that Globus transfers can happen while globus_push.py is still in the queue, thus saving time.
[ ] Provide better logging constructs by extending the use of the summary logs. Currently, summary logs are not properly logged for all parts of the code.
[ ] Determine and code in a better communication pattern between master and slave ranks. Maybe include this in a future BasicFusion Python package?