search

COSIMA / master_index

Scripts to generate COSIMA master index
0 stars 0 forks source link

Temporarily exclude one directory that has a problematic file #33

Closed micaeljtoliveira closed 1 year ago

micaeljtoliveira commented 1 year ago

@aekiss Here is another directory that is making the update fail. This time because of the following file: /g/data/ik11/outputs/access-om2-01/basal_melt_outputs/accessom2-GPC001/rregionocean.nc

with the following error message:

Indexing experiment: basal_melt_outputs
Traceback (most recent call last):
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-22.07/bin/cosima_cookbook-update_db", line 33, in <module>
    sys.exit(load_entry_point('cosima-cookbook==0.7.3', 'console_scripts', 'cosima_cookbook-update_db')())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.07/lib/python3.9/site-packages/cosima_cookbook/database_update.py", line 29, in main
    cc.database.build_index(
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.07/lib/python3.9/site-packages/cosima_cookbook/database.py", line 864, in build_index
    _prune_files(expt, session, files, delete=(prune == "delete"))
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.07/lib/python3.9/site-packages/cosima_cookbook/database.py", line 895, in _prune_files
    oldids = [
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.07/lib/python3.9/site-packages/cosima_cookbook/database.py", line 902, in <listcomp>
    if f.index_time < datetime.fromtimestamp(f.ncfile_path.stat().st_mtime)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.07/lib/python3.9/pathlib.py", line 1232, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/g/data/ik11/outputs/access-om2-01/basal_melt_outputs/accessom2-GPC001/rregionocean.nc'

I suggest we exclude the directory while investigating what is going on with this file, so that we get an updated database over the weekend, as it's been now three weeks since the last successful update. This is assuming that there are no more issues...

micaeljtoliveira commented 1 year ago

BTW I don't think my review is always needed on little things like this (but of course, ask if you want to).

For this PR (and the previous ones), it was more to make sure you were aware of the changes to the directories that are being scanned for new files. But I can also just ping you and merge without asking for a review.

aekiss commented 1 year ago

yep that would be fine

aekiss commented 1 year ago

it has the advantage of not slowing you down if I'm busy with something else

aidanheerdegen commented 1 year ago

Probably worthwhile adding some options to the find command in COSIMA Cookbook to only find readable files

https://github.com/COSIMA/cosima-cookbook/blob/a657fba42d7b599d0e71f589a1867d27e0ab97ca/cosima_cookbook/database.py#L724

Should have thought of that ages ago.

micaeljtoliveira commented 1 year ago

@aidanheerdegen Good idea! I'm planning on spending some time making the indexing more robust and I'll try to implement that.