Open AtefBN opened 1 year ago
It seems that this file (filename = areacella_fx_FGOALS-g2_lgm_r0i0p0.nc
) has no checksum
nor checksum_type
attributes in the synda CMIP5 database you are importing, and those are currently required by esgpull.
The query ran by esgpull on the ESGF search API confirms that the 2 attributes are also missing from the file's metadata: https://esgf-node.ipsl.upmc.fr/esg-search/search?type=File&offset=0&limit=1&format=application%2Fsolr%2Bjson&fields=%2A&query=title%3Aareacella_fx_FGOALS-g2_lgm_r0i0p0.nc&distrib=true&latest=true&retracted=false
I am guessing synda used the same query to fill its database at the time this file was added.
Now if I increase the limit
parameter for this query (numFound
tells us 4 replicas exist in this case), checksum
and checksum_type
do exist in the next 3 replicas' metadata.
Knowing this, 2 things could be done during import to handle missing information:
The first solution might look more complete but it could seriously slow down the import procedure, and does not guarantee missing info will be filled, while the 2nd solution is easy to set up but will definitely introduce divergence between the filesystem and database.
I also encountered this error. My solution was simply to skip the files that were missing metadata bits, and for me, since there weren't a lot of files missing metadata, this was an acceptable loss. I simply added a try/except block with a little extra information to bypass the error halting the program and add the information to the log. I may submit a pull request soon with my proposed code changes:
In esgpull.py
nb_imported = 0
for start in iter_idx_range:
stop = min(len(synda_ids), start + size)
ids = synda_ids[start:stop]
synda_files = synda.scalars(sql.synda_file.with_ids(*ids))
files: list[File] = []
for synda_file in synda_files:
try:
file = synda_file.to_file()
except AttributeError as e:
logger.warning(e)
warn_msg = f"Skipping {synda_file.filename} due to missing database metadata. Continuing to the next file"
print(warn_msg)
logger.warning(warn_msg)
continue
if file.sha not in shas:
file.queries.append(self.legacy_query)
files.append(file)
synda_shas.add(file.sha)
if files:
nb_imported += len(files)
self.db.add(*files)
`(esgpull) -bash-4.2$ esgpull self import-synda /gpfscmip/gpfsdata/esgf/synda-cmn/db/CMIP5/sdt.db Found 810229 files to import, proceed? [y/n]: y Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--AttributeError: 'NoneType' object has no attribute 'upper' See /gpfscmip/gpfsdata/esgf/esgpull1/log/esgpull-import_synda-2023-04-11_08-46-39.log for error log. Aborted! Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- (esgpull) -bash-4.2$ cat /gpfscmip/gpfsdata/esgf/esgpull1/log/esgpull-import_synda-2023-04-11_08-46-39.log [2023-04-11 10:46:46] DEBUG root Locals: { 'self': SyndaFile( file_id=297, url='http://aims3.llnl.gov/thredds/fileServer/cmip5_css01_data/cmip5/output1/LASG-CESS/FGOALS-g2/lgm/fx/atmos/fx/r0i0p0/v20130314/areacella/areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', file_functional_id='cmip5.output1.LASG-CESS.FGOALS-g2.lgm.fx.atmos.fx.r0i0p0.v20130314.areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', filename='areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', local_path='CMIP5/output1/LASG-CESS/FGOALS-g2/lgm/fx/atmos/fx/r0i0p0/v20130314/areacella/areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', data_node='aims3.llnl.gov', checksum=None, checksum_type=None, duration=None, size=42760, rate=None, start_date=None, end_date=None, crea_date='2020-11-03 14:44:18.596992', status='done', error_msg=None, sdget_status=None, sdget_error_msg=None, priority=1000, tracking_id='7181939e-4b39-4eaf-a4be-85eae5b5a9e9', model='FGOALS-g2', project='CMIP5', variable='areacella', last_access_date=None, dataset_id=97, insertion_group_id=1, timestamp='2013-03-12T17:25:11Z' ), 'file_id': 'cmip5.output1.LASG-CESS.FGOALS-g2.lgm.fx.atmos.fx.r0i0p0.v20130314.areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', 'dataset_id': 'cmip5.output1.LASG-CESS.FGOALS-g2.lgm.fx.atmos.fx.r0i0p0.v20130314', 'dataset_master': 'cmip5.output1.LASG-CESS.FGOALS-g2.lgm.fx.atmos.fx.r0i0p0', 'version': 'v20130314', 'master_id': 'cmip5.output1.LASG-CESS.FGOALS-g2.lgm.fx.atmos.fx.r0i0p0.areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', 'url': 'https://aims3.llnl.gov/thredds/fileServer/cmip5_css01_data/cmip5/output1/LASG-CESS/FGOALS-g2/lgm/fx/atmos/fx/r0i0p0/v20130314/areacella/areacella_fx_FGOALS-g2_lgm_r0i0p0.nc', 'local_path': 'CMIP5/output1/LASG-CESS/FGOALS-g2/lgm/fx/atmos/fx/r0i0p0/v20130314/areacella' } Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--
[2023-04-11 10:46:46] ERROR root
Traceback (most recent call last): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/esgpull/tui.py", line 154, in logging yield File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/esgpull/cli/self.py", line 235, in import_synda nb_imported = esg.import_synda(url=path, track=True, ask=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/esgpull/esgpull.py", line 227, in import_synda file = synda_file.to_file() ^^^^^^^^^^^^^^^^^^^^ File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/esgpull/models/synda_file.py", line 64, in to_file checksum_type=self.checksum_type.upper(), ^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'upper' `