Closed jbusecke closed 11 months ago
I found that the amount of simultaneous connections defined here had a big influence on memory when using this on my local laptop (e.g. 10 worked fine, but 50 crashed my machine).
Since I would eventually like to use this within a github action in https://github.com/leap-stc/cmip6-leap-feedstock I think we need to expose this as an option to the user like so:
url_dict = await generate_urls_from_iids(iid_list, connection_limit_per_host = 2)
Now implemented here
I found that the amount of simultaneous connections defined here had a big influence on memory when using this on my local laptop (e.g. 10 worked fine, but 50 crashed my machine).
Since I would eventually like to use this within a github action in https://github.com/leap-stc/cmip6-leap-feedstock I think we need to expose this as an option to the user like so: