search

weech / GRIB.jl

GRIB Interface for Julia
Apache License 2.0
22 stars 2 forks source link

Array Interface? #1

Open rafaqz opened 4 years ago

rafaqz commented 4 years ago

Hi, I was pleased to find this package existed!

At some stage I want to add a GRIB wrapper to GeoData.jl, leveraging this package. It will give a common interface and conversion to NetCDF, GeoTiff, grd and other formats.

It's much easier to write a reliable interface, and do a lot of other things if there is already an Array interface wrapping access to the underlying data as if it is a regular julia array. This can still happen inside a safe function, dispatching on the type that is passed into the do block/anonymous function.

NCDatasets.jl does this quite well for their Variable type: https://github.com/Alexander-Barth/NCDatasets.jl/blob/master/src/NCDatasets.jl#L1063-L1130

Anyway, just a thought if you are doing any more work on this package

Cheers

Edit: in case its not clear, using indexing woudl be to allow loading subsets of the data from disk without loading the whole file, which can be much quicker when you have to load a lot of files.

weech commented 4 years ago

Hi, sorry for the late response. This package is a very thin wrapper around ecCodes, and implementing Array would preclude "loading subsets of the data from disk without loading the whole file". ecCodes provides an iterator interface, so, in order to have random access, you would have to read through the entire file. There might be a way to get around that, but I wouldn't know how.

rafaqz commented 4 years ago

Ok my memory of how grib works is shaky, I've only really used it from Haskel years ago. How are you actually retrieving 2d raster data?

weech commented 4 years ago

I don't know how ecCodes works internally, and the GRIB standard is complicated. The interface ecCodes provides is basically an iterator that returns pointers to message handles, and get/set functions that take a message handle pointer, a string that is the key you want to get/set, and a pointer to an array. To get the 2d raster data, you iterate through the file (using the function codes_grib_handle_new_from_file) until you are at the variable you want, and then call the getter function (codes_get_double_array) with the key "values". The getter function fills a pre-allocated array with the raster data.

Edited with more details