PleiadesLLC4320Model FileNotFoundError

[ah-dinh]

Hi,

I am trying to access LLC4320 outputs on Pleiades using llcreader. There's a file not found error when calling 'llcreader.PleiadesLLC4320Model()'.

Python 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from xmitgcm import llcreader
>>> model = llcreader.PleiadesLLC4320Model()
Traceback (most recent call last):
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/mapping.py", line 135, in __getitem__
    result = self.fs.cat(k)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/spec.py", line 739, in cat
    return self.cat_file(paths[0], **kwargs)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/spec.py", line 649, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/spec.py", line 1009, in open
    f = self._open(
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/implementations/local.py", line 155, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/implementations/local.py", line 250, in __init__
    self._open()
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/implementations/local.py", line 255, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/nobackup/rpaberna/llc/masks/llc_4320_masks.zarr/.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/xmitgcm/llcreader/known_models.py", line 13, in wrapper
    func(*args, **kwargs)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/xmitgcm/llcreader/known_models.py", line 240, in __init__
    super(PleiadesLLC4320Model, self).__init__(store)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/xmitgcm/llcreader/llcmodel.py", line 613, in __init__
    self.masks = self._get_masks()
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/xmitgcm/llcreader/llcmodel.py", line 623, in _get_masks
    zgroup = self.store.open_mask_group()
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/xmitgcm/llcreader/stores.py", line 115, in open_mask_group
    zgroup = zarr.open_consolidated(mapper)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/zarr/convenience.py", line 1178, in open_consolidated
    meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/zarr/storage.py", line 2746, in __init__
    meta = json_loads(store[metadata_key])
  File "/home5/ahdinh/.conda/envs/work/lib/python3.10/site-packages/fsspec/mapping.py", line 139, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

[cspencerjones]

I am also experiencing this issue on Pleiades. Storing these mask files in @rabernat 's nobackup directory is probably unsustainable in the long term. Perhaps we want to ask someone at NASA to host them in a more permanent place (e.g. adjacent to the data, and also perhaps on the ECCO data portal). These are not big files and they are very useful.

[rabernat]

It looks like all my files have disappeared for some reason. I have filed a support ticket with the NAS helpdesk.

[mmazloff]

They have moved where nobackup points Try to find your old one with ls /nobackup*/username Then copy from old to new

[rabernat]

Thanks Matt! They were in /nobackup2/rpaberna. That's so weird.

Is there any alternative to manually copying the data? I've got 20 TB in there...

[mmazloff]

Here is what NAS support says "/nobackupp2 users,

We are beginning the process of decommissioning the /nobackupp2 filesystem. As part of this process, we have created space on the new /nobackupp17 filesystem so you can copy your data there from /nobackupp2.

Please copy your data to the new filesystem by June 3, 2022.

Use the Shift command to copy the data from your /nobackupp2 directory to your directory on the new filesystem. For example:

shiftc -r --hosts=2 /nobackupp2/[username] /nobackupp17/[username]

Important: After you copy your data, submit a ticket to the Control Room and ask them to update your /nobackup/ link to point to the new filesystem.

For information on using the Shift command, see the following article in the HECC Knowledge Base:

Using Shift for Local Transfers and Tar Operations https://urldefense.proofpoint.com/v2/url?u=https-3A__www.nas.nasa.gov_hecc_support_kb_entry_512&d=DwICAg&c=-35OiAkTchMrZOngvJPOeA&r=HSHS34ROdfL7f2oVxuKB761hrMvWN1RWA5k3SHGts_s&m=Of7vKjNyPtVheBSW7GcS7Ga0vhjsfAY7csFRpsn6kdCXlxrkgr1rXP-rFqUdjvJK&s=rJ8kFu0QdJkv1T6kuXxfLGKdpxl1EiMH6p5JgN1DVdg&e=

Note: If you are currently sharing any /nobackupp2 data through the NAS Data Portal service, do not move that data from /nobackupp2.

If you have any questions, or experience any problems, please contact us.

NAS Control Room support@nas.nasa.gov 800 331 8737 650 604 4444"

[rabernat]

My files are now transferred. Can we check whether it is working again?

[ah-dinh]

All is working now. Thank You!