Issues with install raised by pytest

[EdmundBenedict]

Hi, I recently installed xBout on my Ubuntu 18.04 system with pip (and python3.9). I tried to run the Blob2d tutorial, but repeatedly ran into the issue:

`ds = open_boutdataset('./data/BOUT.dmp.*.nc', inputfilepath='./data/BOUT.inp')

Open the data and see what's inside

ds = open_boutdataset('./data/BOUT.dmp.*.nc', inputfilepath='./data/BOUT.inp')

---

KeyError Traceback (most recent call last) ~/.local/lib/python3.9/site-packages/xarray/core/dataset.py in _construct_dataarray(self, name) 1397 try: -> 1398 variable = self._variables[name] 1399 except KeyError:

KeyError: 'PE_XIND'

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last) ~/.local/lib/python3.9/site-packages/xbout/load.py in _infer_contains_boundaries(ds, nxpe, nype) 790 try: --> 791 xproc = int(ds["PE_XIND"]) 792 yproc = int(ds["PE_YIND"])

~/.local/lib/python3.9/site-packages/xarray/core/dataset.py in getitem(self, key) 1501 if hashable(key): -> 1502 return self._construct_dataarray(key) 1503 else:

~/.local/lib/python3.9/site-packages/xarray/core/dataset.py in _constructdataarray(self, name) 1399 except KeyError: -> 1400 , name, variable = _get_virtual_variable( 1401 self._variables, name, self._level_coords, self.dims

~/.local/lib/python3.9/site-packages/xarray/core/dataset.py in _get_virtual_variable(variables, key, level_vars, dim_sizes) 172 else: --> 173 ref_var = variables[ref_name] 174

KeyError: 'PE_XIND'

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last) /tmp/ipykernel_27243/1620813500.py in 1 # Open the data and see what's inside ----> 2 ds = open_boutdataset('./data/BOUT.dmp.*.nc', inputfilepath='./data/BOUT.inp')

~/.local/lib/python3.9/site-packages/xbout/load.py in open_boutdataset(datapath, inputfilepath, geometry, gridfilepath, chunks, keep_xboundaries, keep_yboundaries, run_name, info, **kwargs) 216 if "dump" in input_type: 217 # Gather pointers to all numerical data from BOUT++ output files --> 218 ds, remove_yboundaries = _auto_open_mfboutdataset( 219 datapath=datapath, 220 chunks=chunks,

~/.local/lib/python3.9/site-packages/xbout/load.py in _auto_open_mfboutdataset(datapath, chunks, info, keep_xboundaries, keep_yboundaries, **kwargs) 515 paths_grid, concat_dims = _arrange_for_concatenation(filepaths, nxpe, nype) 516 --> 517 ds = xr.open_mfdataset( 518 paths_grid, 519 concat_dim=concat_dims,

~/.local/lib/python3.9/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combineattrs, **kwargs) 914 closers = [getattr(ds, "_close") for ds in datasets] 915 if preprocess is not None: --> 916 datasets = [preprocess(ds) for ds in datasets] 917 918 if parallel:

~/.local/lib/python3.9/site-packages/xarray/backends/api.py in (.0) 914 closers = [getattr_(ds, "_close") for ds in datasets] 915 if preprocess is not None: --> 916 datasets = [preprocess(ds) for ds in datasets] 917 918 if parallel:

~/.local/lib/python3.9/site-packages/xbout/load.py in _trim(ds, guards, keep_boundaries, nxpe, nype) 755 if any(keep_boundaries.values()): 756 # Work out if this particular dataset contains any boundary cells --> 757 lower_boundaries, upper_boundaries = _infer_contains_boundaries(ds, nxpe, nype) 758 else: 759 lower_boundaries, upper_boundaries = {}, {}

~/.local/lib/python3.9/site-packages/xbout/load.py in _infer_contains_boundaries(ds, nxpe, nype) 796 # numbering scheme 797 # num = nxpei + j, where i={0, ..., nype}, j={0, ..., nxpe} --> 798 filename = ds.encoding["source"] 799 prefix, filenum, extension = Path(filename).suffixes 800 filenum = int(filenum.replace(".", ""))

KeyError: 'source'

`

I then went back and ran pytest --pyargs xbout which raised the errors:

`========================================= warnings summary ========================================== .local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.0.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.3.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.6.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.1.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.4.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.7.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.2.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.5.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_interpolate_parallel_toroidal_points /home/edders/.local/lib/python3.9/site-packages/dask/core.py:115: RuntimeWarning: deallocating CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/tmp/pytest-of-edders/pytest-0/test_data27/BOUT.dmp.8.nc', mode='r', kwargs={'clobber': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}), but file is not already closed. This may indicate a bug. return [_execute_task(a, cache) for a in arg]

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_slab .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_CENTRE] .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_XLOW] .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_YLOW] .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_ZLOW] /home/edders/.local/lib/python3.9/site-packages/xbout/load.py:278: UserWarning: No geometry type found, no physical coordinates will be added warn("No geometry type found, no physical coordinates will be added")

.local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestSave::test_reload_all[toroidal] /home/edders/.local/lib/python3.9/site-packages/xbout/load.py:185: UserWarning: open_boutdataset() called with geometry="toroidal", but we are reloading a Dataset that was saved after being loaded with geometry="toroidal". Applying geometry="toroidal" from the argument. warn(

.local/lib/python3.9/site-packages/xbout/tests/test_plot.py: 24 warnings /home/edders/.local/lib/python3.9/site-packages/xbout/plotting/plotfuncs.py:183: DeprecationWarning: np.int is a deprecated alias for the builtin int. To silence this warning, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations if isinstance(levels, np.int):

.local/lib/python3.9/site-packages/xbout/tests/test_plot.py: 24 warnings /home/edders/.local/lib/python3.9/site-packages/xbout/plotting/plotfuncs.py:194: DeprecationWarning: np.int is a deprecated alias for the builtin int. To silence this warning, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations if isinstance(levels, np.int):

-- Docs: https://docs.pytest.org/en/stable/warnings.html ====================================== short test summary info ====================================== FAILED .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_slab FAILED .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_CENTRE] FAILED .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_XLOW] FAILED .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_YLOW] FAILED .local/lib/python3.9/site-packages/xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_ZLOW] ================ 5 failed, 230 passed, 507 skipped, 63 warnings in 296.11s (0:04:56) ================ `

I have repeatedly reinstalled the code, recompiled python, re-ran the Blob2D example (before I realised that this error was deeper), but I am slightly lost as to what is wrong. Is there anything else I should be doing? I can copy in more of the pytest output if that would help? Thank you very much, Edmund

[johnomotani]

The pytest failures are probably just the version incompatibility that was fixed in #209, which only affects the integration methods so shouldn't cause the original problem you reported - I should probably make a new bug-fix release soon though.

Which version of BOUT++ did you use to generate the blob2d data?

[EdmundBenedict]

I used the docker image to run this simulation: boutproject/bout-next:9f4c663-sundials-ubuntu Although I previously had the same issue with another image: boutproject/bout-next:9f4c663-petsc

[johnomotani]

Ah, that docker image is very old (~3 years) and seems to be from before the 4.2.0 release. We'd like to support data from older releases with xBOUT, but due to limited developer time we don't test them and won't usually work on bugs that only affect versions <4.3.0 (PRs would be welcome to work around issues like the one you have run into though!).

There are a couple of more recent Docker builds (see https://hub.docker.com/r/boutproject/bout-next/tags/?page=1&ordering=last_updated), which should probably work OK, but the Docker builds are only updated sporadically, so I'd suggest using the Github version (see instructions at https://bout-dev.readthedocs.io/en/latest/user_docs/installing.html#obtaining-bout) in order to have all the latest bug fixes, etc. I'm actually not sure BOUT++ should have Docker as the first entry on the getting started page - going to add this to the discussion on https://github.com/boutproject/BOUT-dev/issues/2186...

[johnomotani]

I guess it should also be possible to update the version of BOUT++ in the Docker image, although I'm not sure exactly what state the git repo is in in the Docker images. Something like

cd BOUT-dev
git checkout master
git pull

might work? Although it would also need a reconfigure and rebuild afterwards...

[EdmundBenedict]

Thank you very much for the reply. I'll try generating the Blob2d output with the newer docker image now, but I may have issues installing it from the source code: I tried this before but found that there were issues in the compilation from the lack of a set of solvers, Sundial, from LLNL. When I tried to download and compile these solvers I found that the version of cmake required to compile them was only available on the newer Ubuntu version, 20.04. I'll take a look at this again though, maybe I made some mistake

[johnomotani]

Since you use pip - it is possible to install a more recent version of cmake using, e.g.

pip install --user cmake

[EdmundBenedict]

Hi, I have downloaded and compiled the BOUT++ release version 4.4.0 and re-run the tutorial. It now mostly runs fine, the only (minor) issue being that the references to ds['t_array'] raise issues, but if these are changed to ds['t'] the code works fine. I'm not sure if the label for the time array changed between versions or what. Thank you very much for your help, Edmund