Windows test failures for test_accessor_sliced_datacube and test_grdview_drapegrid_dataarray

weiji14 commented 2 years ago

Description of the problem

Two unit tests have been failing consistenly on the Windows CI tests (both the regular and Dev versions). See e.g. https://github.com/GenericMappingTools/pygmt/runs/7890620920?check_suite_focus=true#step:11:661

Full code that generated the error

First test failure is due to this code, in particular, the os.remove line

https://github.com/GenericMappingTools/pygmt/blob/58a9b30290a6640b8110be58947ced9d9e80d0c0/pygmt/tests/test_accessor.py#L77-L100

Second test failure is due to some mismatch between baseline and actual output image

Full error message

================================== FAILURES ===================================
________________________ test_accessor_sliced_datacube ________________________

    @pytest.mark.skipif(
        gmt_version < Version("6.4.0"),
        reason="Upstream bug fixed in https://github.com/GenericMappingTools/gmt/pull/6615",
    )
    def test_accessor_sliced_datacube():
        """
        Check that a 2D grid which is sliced from an n-dimensional datacube works
        with accessor methods.

        This is a regression test for
        https://github.com/GenericMappingTools/pygmt/issues/1578.
        """
        try:
            fname = which(
                "https://github.com/pydata/xarray-data/raw/master/eraint_uvz.nc",
                download="u",
            )
            with xr.open_dataset(fname) as dataset:
                grid = dataset.sel(level=500, month=1, drop=True).z

            assert grid.gmt.registration == 0  # gridline registration
            assert grid.gmt.gtype == 1  # geographic coordinate type
        finally:
>           os.remove(fname)
E           PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'eraint_uvz.nc'

C:\Miniconda3\envs\pygmt\lib\site-packages\pygmt\tests\test_accessor.py:100: PermissionError
---------------------------- Captured stderr call -----------------------------
Warning: WARNING]: No 3-D array in file D:\a\pygmt\pygmt\tmp-test-dir-with-unique-name\eraint_uvz.nc.  Selecting first 3-D slice in the 4-D array z
______________________ test_grdview_drapegrid_dataarray _______________________
Error: Image files did not match.
  RMS Value: 51.91310207540863
  Expected:  
    D:\a\pygmt\pygmt\tmp-test-dir-with-unique-name\results\pygmt.tests.test_grdview.test_grdview_drapegrid_dataarray\baseline.png
  Actual:    
    D:\a\pygmt\pygmt\tmp-test-dir-with-unique-name\results\pygmt.tests.test_grdview.test_grdview_drapegrid_dataarray\result.png
  Difference:
    D:\a\pygmt\pygmt\tmp-test-dir-with-unique-name\results\pygmt.tests.test_grdview.test_grdview_drapegrid_dataarray\result-failed-diff.png
  Tolerance: 
    2

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.7.1.dev43+g58a9b302
System information:
  python: 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 06:57:19) [MSC v.1929 64 bit (AMD64)]
  executable: C:\Miniconda3\envs\pygmt\python.exe
  machine: Windows-10-10.0.20348-SP0
Dependency information:
  numpy: 1.23.2
  pandas: 1.4.3
  xarray: 2022.6.0
  netCDF4: 1.6.0
  packaging: 21.3
  geopandas: 0.11.1
  ghostscript: 9.54.0
  gmt: 6.4.0
GMT library information:
  binary dir: C:/Miniconda3/envs/pygmt
  cores: 2
  grid layout: rows
  library path: C:/Miniconda3/envs/pygmt/Library/bin/gmt.dll
  padding: 2
  plugin dir: C:/Miniconda3/envs/pygmt/Library/bin/gmt_plugins
  share dir: C:/Miniconda3/envs/pygmt/Library/share/gmt
  version: 6.4.0

seisman commented 2 years ago

For the 2nd test failure, the equivalent PyGMT script is:

import pygmt
from pygmt.helpers.testing import load_static_earth_relief
grid = load_static_earth_relief()
xrgrid = pygmt.grdcut(grid=grid, region=[-55, -50, -18, -12])
drapegrid = 1.1 * xrgrid
fig = pygmt.Figure()
fig.grdview(grid=xrgrid, drapegrid=drapegrid, cmap="oleron", surftype="c")
fig.savefig("pygmt-test.png")

And the equivalent GMT bash script is:

gmt grdcut @static_earth_relief.nc -R-55/-50/-18/-12 -Ggrid.nc
gmt grdmath grid.nc 1.1 MUL = drape.nc
gmt grdview grid.nc -Gdrape.nc -Coleron -Qs -png gmt-test

The outputs are different on macOS (ignoring the colorbar which is automatically added in one-liner mode):	bash GMT	PyGMT

Our baseline image is the same as the right-side image, and the resulting image on Windows looks the same as the left-side image. So, I think:

the baseline image for test_grdview_drapegrid_dataarray is wrong
after correcting the baseline image, the test is likely to pass on Windows but fail on Linux and macOS, which may point to an upstream bug

seisman commented 2 years ago

Both failures are not easy to debug and are likely due to upstream bugs.

They are tracked by PR https://github.com/GenericMappingTools/pygmt/pull/2068 and https://github.com/GenericMappingTools/pygmt/pull/2073.

I think we should mark the two tests as xfail temporarily (https://github.com/GenericMappingTools/pygmt/pull/2075) so that all checks are green.

seisman commented 1 year ago

@weiji14 I believe the test_accessor_sliced_datacube failure on Windows is because GMT forgets to close the datacube NetCDF file after reading it. The bug was fixed in https://github.com/GenericMappingTools/gmt/pull/7573.

Could you please trigger a new dev build on conda-forge so that we can verify it?

weiji14 commented 1 year ago

Ok, new GMT 6.5.0.dev6+95c3431 build at https://github.com/conda-forge/gmt-feedstock/pull/254.

weiji14 commented 1 year ago

Ok, test_accessor_sliced_datacube has been fixed in #2073 when using GMT 6.5.0, still need to fix test_grdview_drapegrid_dataarray in #2068.

GenericMappingTools / pygmt

Windows test failures for test_accessor_sliced_datacube and test_grdview_drapegrid_dataarray #2062