search

jbusecke / pangeo-forge-esgf

Using queries to the ESGF API to generate urls and keyword arguments for receipe generation in pangeo-forge
Apache License 2.0
6 stars 4 forks source link

Expose `limit_per_host` to user #16

Closed jbusecke closed 11 months ago

jbusecke commented 12 months ago

I found that the amount of simultaneous connections defined here had a big influence on memory when using this on my local laptop (e.g. 10 worked fine, but 50 crashed my machine).

Since I would eventually like to use this within a github action in https://github.com/leap-stc/cmip6-leap-feedstock I think we need to expose this as an option to the user like so: 

url_dict = await generate_urls_from_iids(iid_list, connection_limit_per_host = 2)
jbusecke commented 11 months ago

Now implemented here