search

mjwoods / RNetCDF

Read and write netcdf format in R
Other
24 stars 9 forks source link

Avoid empty vectors in output of `var.inq.nc` #60

Closed mjwoods closed 5 years ago

mjwoods commented 5 years ago

Recent versions of netcdf provide an interface to the pluggable filter mechanism in hdf5, mainly to support advanced compression methods.

RNetCDF reports the filter details in the output of var.inq.nc. The filter is specified by filter_id, and arguments to the filter function are shown in filter_params. The filter_params are passed to netcdf as an integer vector, so integer(0) implies that the filter function requires no arguments.

Some existing user code does not expect empty vectors in the output from var.inq.nc. A possible solution is to translate integer(0) to NA in var.inq.nc, and vice-versa in var.def.nc.

mjwoods commented 5 years ago

Several other items can potentially have length 0 in the output of var.inq.nc. For example, chunksizes is NULL for contiguous storage. Is this a problem, @dblodgett-usgs ? If not, do we need to worry about integer(0) in filter_params?

dblodgett-usgs commented 5 years ago

We added a little logic to deal with it, so this should be fine.

ckluss commented 2 years ago

is this related to this issue?

https://github.com/hypertidy/ncmeta/issues/42

mjwoods commented 2 years ago

Hi @ckluss , I don't think it's the same issue. In the case you are referring to, the filter parameters seem to be integer vectors of length 1. The problem here was about handling zero-length vectors. In the end, we left them as integer(0).

mdsumner commented 2 years ago

thanks @mjwoods - would it make sense for scalar dimids to be length zero? I find 'NA' a bit funny in that case, but appreciate there might be more to it

mjwoods commented 2 years ago

I agree that NA is awkward, but it is a legacy from the early days of RNetCDF. Changing it now could break a lot of user code.

mdsumner commented 2 years ago

no problem 👌

ckluss commented 2 years ago

thanks @mjwoods but it seems to be a RNetCDF related problem? I'm completely new to the field.

mjwoods commented 2 years ago

That's debatable @ckluss . RNetCDF added a feature about a year ago that appears to have broken ncmeta when reading certain files. We have chosen to fix the problem with changes to ncmeta, which should have fewer side-effects than further changes to RNetCDF.

mdsumner commented 2 years ago

yes agreed, I will catch more of this information at a later stage when I can really focus on it - we can have nested data frames, for example