esgf2-us / intake-esgf

Programmatic access to the ESGF holdings
https://intake-esgf.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
9 stars 5 forks source link

File info for Solr indices can fail if request is too large #41

Open nocollier opened 3 months ago

nocollier commented 3 months ago

In order to reduce the number of requests and therefore overall time, I reworked the get_file_info to get all info for all datasets in your search. The globus search can handle this. However, if you have a lot of dataset_ids, this appears to violate the RESTapi max length and so all Solr nodes fail to return a response.