CI workflows / test setup: netCDF test files not detected

[sadielbartholomew]

Some recent CI jobs run via GitHub Actions have shown several errors that are related through erroring on missing files, namely one of a set of three netCDF testing files:

• DSG_timeSeries_contiguous.nc;
• DSG_timeSeries_indexed.nc;
• gathered.nc.

(mostly ubuntu-latest OS jobs, though I strongly suspect that's because those jobs run fastest so fail earliest, cancelling the jobs running on other OS's). I don't see these errors locally.

I suspect something is wrong with the workflow configuration such that these files aren't created as they should be, since they get generated by create_test_files.py. (I'm not looking into this with any sort of priority at the moment, but will eventually get it sorted!).

Details of test errors from CI jobs

The reports show something very similar to this:

======================================================================
ERROR: test_Data__init__compression (__main__.DataTest)
Test Data initialised from compressed data sources.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 2837, in test_Data__init__compression
    for f in cfdm.read("DSG_timeSeries_contiguous.nc"):
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file DSG_timeSeries_contiguous.nc

======================================================================
ERROR: test_Data_compressed_array (__main__.DataTest)
Test the `compressed_array` Data property.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 4044, in test_Data_compressed_array
    f = cfdm.read("DSG_timeSeries_contiguous.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file DSG_timeSeries_contiguous.nc

======================================================================
ERROR: test_Data_get_compressed (__main__.DataTest)
Test the Data methods which get compression properties.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 4076, in test_Data_get_compressed
    f = cfdm.read("DSG_timeSeries_contiguous.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line [309](https://github.com/NCAS-CMS/cf-python/actions/runs/3410953930/jobs/5674553501#step:13:310), in read
Run date: 2022-11-07 14:02:20.311924
    raise IOError(f"Can't read non-existent file {filename}")
Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.31 
HDF5 library: 1.12.2 
netcdf library: 4.9.0 
udunits2 library: /usr/share/miniconda3/envs/cf-latest/lib/libudunits2.so.0 
Python: 3.9.13 /usr/share/miniconda3/envs/cf-latest/bin/python
netCDF4: 1.6.1 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/netCDF4/__init__.py
cftime: 1.6.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/cftime/__init__.py
numpy: 1.23.4 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/numpy/__init__.py
psutil: 5.9.3 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/psutil/__init__.py
scipy: 1.9.3 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/scipy/__init__.py
matplotlib: 3.6.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/matplotlib/__init__.py
ESMF: 8.2.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/ESMF/__init__.py
cfdm: 1.10.0.1 /home/runner/work/cf-python/cf-python/cfdm/cfdm/__init__.py
cfunits: 3.3.5 /usr/share/miniconda3/envs/cf-latest/lib/python3.9/site-packages/cfunits/__init__.py
cfplot: not available 
cf: 3.14.0b0 /home/runner/work/cf-python/cf-python/main/cf/__init__.py

OSError: Can't read non-existent file DSG_timeSeries_contiguous.nc

======================================================================
ERROR: test_Data_get_count (__main__.DataTest)
Test the `get_count` Data method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 4130, in test_Data_get_count
    f = cfdm.read("DSG_timeSeries_contiguous.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file DSG_timeSeries_contiguous.nc

======================================================================
ERROR: test_Data_get_index (__main__.DataTest)
Test the `get_index` Data method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 4143, in test_Data_get_index
    f = cfdm.read("DSG_timeSeries_indexed.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file DSG_timeSeries_indexed.nc

======================================================================
ERROR: test_Data_get_list (__main__.DataTest)
Test the `get_list` Data method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 4156, in test_Data_get_list
    f = cfdm.read("gathered.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file gathered.nc

======================================================================
ERROR: test_Data_uncompress (__main__.DataTest)
Test the `uncompress` Data method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cf-python/cf-python/main/cf/test/test_Data.py", line 3923, in test_Data_uncompress
    f = cfdm.read("DSG_timeSeries_contiguous.nc")[0]
  File "/home/runner/work/cf-python/cf-python/cfdm/cfdm/read_write/read.py", line 309, in read
    raise IOError(f"Can't read non-existent file {filename}")
OSError: Can't read non-existent file DSG_timeSeries_contiguous.nc

----------------------------------------------------------------------
Ran 131 tests in 18.185s

FAILED (errors=7, skipped=2)

[davidhassell]

Thanks for chasing this up, Sadie.

[sadielbartholomew]

OK I've realised why this is happening now. I actually fixed this issue in a step in a separate workflow already, with this step:

https://github.com/NCAS-CMS/cf-python/blob/92048a0f56c97a1df0216cb6b94f34325d6cb64f/.github/workflows/run-test-suite.yml#L117-L123

I think I left it out of the workflow for dask-migration-testing.yml because I didn't think we used the files that get created in test_Data, which is clearly wrong in hindsight...

I'll get the fix incorporated into the dask-migration-testing.yml now and close this once I can show it is fixed.

[sadielbartholomew]

This was fixed by #535 but not automatically closed by GitHub due to not being a PR to the master branch. Closing now.