globus-gladier / gladier

An SDK for rapidly developing Globus Flows while leveraging Globus Compute
Apache License 2.0
9 stars 3 forks source link

Test config files under a multi-processing environment #247

Closed NickolausDS closed 1 year ago

NickolausDS commented 1 year ago

When doing a lot of runs with multiple processes, multiple entities are potentially writing to the config file at the same time, which adds the possibility for corruption. Investigate this further and add any changes required to mitigate the problem.

NickolausDS commented 1 year ago

Tested locally with both multithreading and multiprocessing scripts and found no issues that caused problems. It's still possible multiprocessing problems exist and we'll need to split out token handling into a separate DB like the Globus Compute folks did, but for now I think this should work fine.