search

hainegroup / oceanspy

A Python package to facilitate ocean model data analysis and visualization.
https://oceanspy.readthedocs.io
MIT License
98 stars 32 forks source link

Kernel crashes when attempting to load LLC4320 through Scisever #309

Closed Mikejmnez closed 1 year ago

Mikejmnez commented 1 year ago

Description

There is some missing data in some zarr files associated with vertical velocity, which causes (chuncking and performance) errors when creating the OceanDataset in the following way:

od = oceanspy.open_oceandataset.from_catalog('LLC4320')

You get something like:

/home/idies/miniconda3/envs/Oceanography/lib/python3.9/site-packages/xarray/core/indexing.py:1228: PerformanceWarning: Slicing is producing a large chunk. To accept the large
chunk and silence this warning, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):
    ...     array[indexer]

To avoid creating the large chunks, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):
    ...     array[indexer]
  return self.array[key]
/home/idies/miniconda3/envs/Oceanography/lib/python3.9/site-packages/dask/array/core.py:4781: PerformanceWarning: Increasing number of chunks by factor of 24
  result = blockwise(
/home/idies/miniconda3/envs/Oceanography/lib/python3.9/site-packages/dask/array/core.py:4781: PerformanceWarning: Increasing number of chunks by factor of 13
  result = blockwise(
/home/idies/miniconda3/envs/Oceanography/lib/python3.9/site-packages/xarray/core/indexing.py:1228: PerformanceWarning: Slicing is producing a large chunk. To accept the large
chunk and silence this warning, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):
    ...     array[indexer]

and then, the kernel crashes. I wasn't getting this error through datascope, so it took me a while to figure this one out!

Solution

The problem arose when I added the vertical velocity to the preexisting velocity zarr stores (files). My error was that I missed the last day of data and so the last three zarr files do not have the vertical velocity. Forcing files are also missing for the last day, but that didin't cause any error (because forcing files are separate).

No PR is needed to fix this. I just have to transform and add the missing files from binary to zarr. This should be done later today, and once all is good, I will close this issue.

Mikejmnez commented 1 year ago

Finished transforming the remaining binary w files. The od loads fine. Therefore, I am closing issue.