floriankrb
commented
6 years ago From chatting with @mraspaud and other :
- Satpy does not handle my Level 2/3 MSG/EPS/MODIS data. But it would be easy to add it (clean plugin API for additional readers). Satpy is about providing a unified way to deal with satellite data. Perhaps we will choose this way and add our readers to satpy.
- big images (bigger than 20k x 20k may be difficult to handle with pyresample). Most dask improvements are in the develop branch for now.
I believe I could reuse pyresample, satpy or other packages in pytroll, but I need some guidance.
I am working on generating satellite albedo products and I need to process many (all?) available satellite data, Level 2 and Level 3 (full archives), reproject them on a common grid, compare, etc.
More precisely, we want to read, preprocess, reproject to a common grid the METOP-AVHRR data, MSG-SEVIRI, MODIS collection 5 and 6, and SPOT-VGT albedo datasets; the grids being either
Then, we want to stack the harmonised data through time and store these datacubes efficiently to be able to read them rather quickly when we need to compare/plot results. We plan to use the xarray/dask?/pandas/seaborn/bokeh/datashader?/holoviews ecosystem.
In order to have a generic tool allowing us to project from whatever grid to any other grid, we have started to use pyresample.
1) So far, I’ve been struggling with grid and projection definitions for sinusoidal and geostationary products like those from METOP & MSG. Are these grids already defined somewhere in the satpy code ?
2) Also we plan to suffer on the optimisation of data reading and storage optimisation for such big data files as the METOP (18001 x 36000 pixels, full globe 1km) & MODIS global datasets. How far are you in using dask to improve this ?
3) We also find out about the satpy package you’re developing, and which seems to offer readers for most meteo satellites available. I do not have yet a very clear views of the possibilities offered by Satpy and was wondering if you think that the tools you propose could be directly used for our objectives or if it is preferable for us to either use some specific functions of the package (like pyresample, and probably others), and maybe borrow some other code here and there, to develop our own tool. If so, perhaps could you point me to the satpy functions that could be of interest for us. Basically, is it a good use case fo satpy : should I reuse satpy as a user or go in the code to steal the logic ?
Would you rather chat on slack or should I ask question on stackoverflow or open a github issue or call by phone ?
Please do not hesitate to tell me that you are busy and don't have time for this.