AWS NOAA WHOI

[kathrynberger]

A recipe for AWS NOAA WHOI Sea Surface Temperature, one of the three resources made available as part of NOAA's Oceanic Climate Data Records (see: https://registry.opendata.aws/noaa-cdr-oceanic/)

File pattern identified and tested both using pruned recipe feature, as well as running on three months worth of data. Output looks correct and as expected.

Closes out issue: https://github.com/developmentseed/aws-asdi/issues/21

[cisaacstern]

/run aws-noaa-sea-surface-temp-whoi

[pangeo-forge[bot]]

The test failed, but I'm sure we can find out why!

Pangeo Forge maintainers are working diligently to provide public logs for contributors. That feature is not quite ready yet, however, so please reach out on this thread to a maintainer, and they'll help you diagnose the problem.

[rbavery]

@kathrynberger Could this be failing because there needs to be a requirements.txt in the recipe folder? for s3fs ? I needed to install that locally to get the recipe to run. I've seen some other recipes include a requirements.txt, like this one https://github.com/pangeo-forge/staged-recipes/pull/220/files

if this is the issue, maybe it'd be good to include adding an optional requirements.txt as a step in https://pangeo-forge.readthedocs.io/en/latest/pangeo_forge_cloud/recipe_contribution.html ?

[cisaacstern]

Thanks for jumping in @rbavery!

Gosh, I've really got to resolve https://github.com/pangeo-forge/pangeo-forge-orchestrator/pull/150, without which it's basically impossible for community members to debug these errors. 🙃

So the cloud workers definitely have s3fs by default. A requirements.txt would be for more exotic requirements. (Unless something really major has changed in the last month that I missed!)

Looking at the backed logs, I'm seeing:

File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/reference_hdf_zarr.py", line 30, in scan_file
    with file_opener(fname, **config.netcdf_storage_options) as fp:
  File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 283, in file_opener
    with opener as fp:
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/core.py", line 103, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/spec.py", line 1094, in open
    f = self._open(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/local.py", line 175, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/local.py", line 273, in __init__
    self._open()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/local.py", line 278, in _open
    self.f = open(self.path, mode=self.mode)
RuntimeError: FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/noaa-cdr-sea-surface-temp-whoi-pds/data/1988/SEAFLUX-OSB-CDR_V02R00_SST_D19880102_C20160820.nc' [while running 'Start|scan_file|Reshuffle_000|finalize|Reshuffle_001/scan_file/Execute-ptransform-56']
"

[sharkinsspatial]

/run aws-noaa-sea-surface-temp-whoi

[pangeo-forge[bot]]

:tada: The test run of aws-noaa-sea-surface-temp-whoi at 166954e3d84a865f75f3a3abd1f31522b1d082cc succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1389/aws-noaa-sea-surface-temp-whoi.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

[cisaacstern]

@sharkinsspatial reports that the test data looks good, so I'll merge this.