Avoid using h5netcdf?

[JackKelly]

MetOfficeMessage.load_netcdf() ends with the line:

return xr.open_dataset(netcdf_bytes_io, engine='h5netcdf')

This means we must include h5netcdf in environment.yml. But, including h5netcdf forces conda to downgrade cartopy from v0.18 to v0.17, which breaks nwp_plot.py:

>       ax = plt.axes(projection=ccrs.OSGB(approx=True))
E       TypeError: __init__() got an unexpected keyword argument 'approx'

metoffice_ec2/nwp_plot.py:35: TypeError

One solution may be to see if we can use an xarray engine which isn't h5netcdf.

[JackKelly]

It's possible that we can just remove engine='h5netcdf'. Or see here for an alternative approach: https://github.com/pydata/xarray/issues/1075#issuecomment-373541528

[flowirtz]

Getting a lot of these errors atm:

2020-05-29 17:49:47,688 - metoffice_ec2 - ERROR - No module named 'h5netcdf'
Traceback (most recent call last):
  File "scripts/ec2.py", line 114, in loop
    load_subset_and_save_data(mo_message, s3)
  File "scripts/ec2.py", line 64, in load_subset_and_save_data
    dataset = mo_message.load_netcdf()
  File "/usr/src/app/metoffice_ec2/message.py", line 73, in load_netcdf
    return xr.open_dataset(netcdf_bytes_io)
  File "/opt/conda/envs/metoffice_ec2/lib/python3.8/site-packages/xarray/backends/api.py", line 533, in open_dataset
    store = backends.H5NetCDFStore.open(
  File "/opt/conda/envs/metoffice_ec2/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 123, in open
    import h5netcdf
ModuleNotFoundError: No module named 'h5netcdf'

Reopening this issue, didn't seem to resolve it.

[flowirtz]

Reverted this via #30 to fix deployment.

[JackKelly]

Eak, sorry about this, I really need to write some more tests so we can catch simple failures like this before pushing to master. Sorry, my bad! I'll look into this next week

[flowirtz]

No worries! Yes, we should probably add some tests haha

[JackKelly]

I've recreated this issue locally.

I've tried changing the engine to all the options (other than h5netcdf) and it always responds: ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

[JackKelly]

I've recreated this issue locally.

I've tried changing the engine to all the options (other than h5netcdf) and it always responds: ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

But I think I've got it working without h5netcdf using the netCDF4 trick described here: https://github.com/pydata/xarray/issues/1075#issuecomment-373541528

@FWirtz no rush, but please could you try this new code on AWS?

[flowirtz]

@JackKelly Will do! Thanks.

[flowirtz]

That worked!