GSTT-CSC / MLOps

Framework for building ML apps
GNU General Public License v3.0
9 stars 5 forks source link

parallelise the xnat_build_dataset #120

Closed laurencejackson closed 1 year ago

laurencejackson commented 1 year ago

This is a bottleneck when datasets are large, should be able to parallelise.

See here for example of async requests. https://medium.com/swlh/parallel-asynchronous-api-call-in-python-f6aa663206c6

See here for example for how to share sessions across workers with JSESSION https://xnat.readthedocs.io/en/latest/static/tutorial.html#re-using-xnat-jsession

laurencejackson commented 1 year ago

resolved by #121