Open jbusecke opened 3 months ago
Yup I just confirmed that the distributed search does not work properly 😡:
from pangeo_forge_esgf.client import ESGFClient
iid = 'CMIP6.CMIP.MPI-M.MPI-ESM1-2-HR.historical.r1i1p1f1.Amon.tas.gn.v20190710'
for search_node in ["https://esgf-node.llnl.gov",
"https://esgf-data.dkrz.de",
"https://esgf.nci.org.au",
"https://esgf-node.ornl.gov",
"https://esgf-node.ipsl.upmc.fr",
"https://esg-dn1.nsc.liu.se",
"https://esgf.ceda.ac.uk",]:
client = ESGFClient(search_node, distributed=True)
dataset_id = client.get_instance_id_input([iid])[iid]['id']
details = client._search_files_from_dataset_ids([dataset_id])
print(f"{search_node=} {[i['id'] for i in details]}")
Which means I will now have to query every index node separately and combine results, what a pain.
Ill stop here for now, but list the options I have going forwards
Loop over multiple index nodes as part of my client class (seems slow and annoying)
Use intake-esgf which already does this (would reduce my maintenance burden, but probably also be very slow; needs testing)
Rewrite the client AGAIN to do all the requesting async (probably fastest, but also significant work)
Seems useless, since I might as well go to 2. So I guess ill time 2. and then decide if it might be worth embarking on 3.
😩
Example how to maybe use intake-esgf:
!pip install git+https://github.com/jbusecke/intake-esgf.git@http-links
import intake_esgf
from intake_esgf import ESGFCatalog
from intake_esgf.base import NoSearchResults
from pangeo_forge_esgf.utils import facets_from_iid
intake_esgf.conf.set(indices={
"esgf-node.llnl.gov":True,
"esg-dn1.nsc.liu.se":True,
"esgf-data.dkrz.de":True,
"esgf-node.ipsl.upmc.fr":True,
"esgf-node.ornl.gov":True,
"esgf.ceda.ac.uk":True,
# "esgf.nci.org.au":True,
})
cat = ESGFCatalog()
def get_urls_from_intake_esgf(iid:str, cat:ESGFCatalog):
print(iid)
facets = facets_from_iid(iid)
facets['version'] = facets['version'].replace('v','') # shouldn't be necessary once https://github.com/jbusecke/pangeo-forge-esgf/pull/41 is merged
try:
res = cat.search(**facets)
return res.to_http_link_dict()
except NoSearchResults:
return None
iid = 'CMIP6.CMIP.MPI-M.MPI-ESM1-2-HR.historical.r1i1p1f1.Amon.tas.gn.v20190710'
a = get_urls_from_intake_esgf(iid, cat)
[i['path'] for i in a]
Ah here is a way to fail out these instances of incomplete filenames:
from pangeo_forge_esgf.client import ESGFClient
import json
iid = 'CMIP6.CMIP.MPI-M.MPI-ESM1-2-HR.historical.r1i1p1f1.Amon.tas.gn.v20190710'
client = ESGFClient()
d = client.get_instance_id_input([iid])
print(json.dumps(d, indent=4))
This produces
My idea is to use "datetime_start": "1975-01-16T12:00:00Z", "datetime_stop": "2014-12-16T12:00:00Z",
inject them as dataset attributes, and then run a check against the actual dataset time data to confirm that the dataset covers this (or at least close to this).
https://github.com/leap-stc/cmip6-leap-feedstock/issues/116#issuecomment-2101604165 describes a case where I get a nice list of files back, but they are not complete! How do we detect this case before ingesting?