search

jgomezdans / KaFKA

Kascading Fast Kalman Assimilation
GNU General Public License v3.0
5 stars 8 forks source link

Optimal I/O #2

Open jgomezdans opened 7 years ago

jgomezdans commented 7 years ago

Currently, the I/O of the observations and subsequent results are just arrays. This is far form ideal, as they are created and take up memory lspace and thus reduce the size of problems that can be solved. A better approach is to have a dedicated method to process the observations on a per timestep/band basis, and use some specialist method to write out the data to an efficient format (HDF5, netCDF, whatever).

jgomezdans commented 7 years ago

So very tempted to use kea file format: HDF5, easy to use, can be read by GDAL, but doesn't need GDAL (see geospatial-hdf5), can probably be converted easily to netcdf, but mostly just needs h5py and a few python only dependencies.

jgomezdans commented 7 years ago

So not sure KEA is the right format to use here. Plain HDF5 seems best, but maybe netcdf4 is best, re-using stuff from eoldas_ng_observations....

jgomezdans commented 7 years ago

So I decided to go for netCDF4, as I had some code that could be scavenged. In 2e709be7cdecdda4cce0291153f04b409318c24e I have introduced a class to do this, but it needs a bit of clarification on how to use it in practice.

jgomezdans commented 7 years ago

OK, so the netcdf4 stuff has issues:

NPounder commented 7 years ago

I will take a look at that

jgomezdans commented 7 years ago

Cool. The main issue why this isn't working is some bug in my netcdf code. We ought to be able to create a netcdf file from an e.g. MODIS (or "master") file so that the created file takes the geotransform, projection etc from the master file. 

from utils import OutputFile

fname = "/tmp/MOD09GA.A2006212.h11v04.006.2015120014933.hdf"

s = OutputFile("testme.nc", 
    input_file='HDF4_EOS:EOS_GRID:"%":MODIS_Grid_500m_2D:sur_refl_b02_1' % fname)

Should in theory work, but it crashes for some reason.

NPounder commented 7 years ago

Great, I should be able to work with that.

jgomezdans commented 7 years ago

OK, since you're at it ;) Can you

?

NPounder commented 7 years ago

On it.

NPounder commented 7 years ago

I've added a very basic function that adds a time slice to an existing variable and checked that it does do that. Currently there is very little error checking/handling. Is this the kind of thing you wanted?

I've not written anything to create an empty variable, you could use create_variable for the first slice and then add afterwards.

It assumes a single time step added at a time. I think I should modify it to allow for adding multiple time-steps at once. That wouldn't be difficult.

I can open the files with netcdf4 or ncdump after they have been created but I'm not so confident with gdal so haven't figured that out yet...

NPounder commented 7 years ago

Now updated so you can add a single or multiple timesteps. Also checks dimensions match and raises errors if not. Can use to add first time-step as long as variable has been created

NPounder commented 7 years ago

Added method to create empty variable.

Update variable designed to only work for variables with time dimension. Other variables should be filled with the create_variable method and not create_empty_variable method