native python readers for GOES GVAR and McIDAS Area format

[jferencik]

Feature Request

This is more of a suggestion/question than a feature request. I wanted to ask if adding a native python readers for GOES GVAR and McIDAS Area to pysat would be of interest to anyone.

Is your feature request related to a problem? Please describe. GOES GVAR is a low level binary format used by pre ABI GOES satellites. To my knowledge there are no free/open source readers available in python. There is a reader written in C that I am aware of however. The reader is much slower than the C implementation, nevertheless, it can be easily integrated into python based processing pipelines.

The MciDAS area format is merely a port of the McIDAS Area Java code from SSEC ( specially the navigation code ) It is available at

https://github.com/jferencik/pyarea

Describe the solution you'd like I have developed a native python GOES GVAR reader. I have also ported the GOES Fortran navigation to python (ELUG).

The reader was tested on 20 years worth of data for both, GOES East and West and has failed to read a handful of images only.

Describe any changes to existing user workflow I amnot sure how this integration would be done, GVAR is quite different than most of the availble formats.

Additional context Have you considered any alternative solutions or is there anything else that would help describe your request.

[djhoese]

Wow @jferencik, nice work. I work at the SSEC right next to some of the McIDAS devs and users. I don't personally work on the project though. Use of area files would be a very interesting feature indeed. Depending on what metadata is kept in them it could be possible to make a reader for them.

For GVAR, I'm not familiar with the format, but it sounds like you have concerns about it fitting in to satpy's model. What kind of concerns do you have? What about GVAR does not fit with how we do things now? If you're not sure, could you describe the format and how it differs from existing formats like VIIRS SDRs or the GOES-16 ABI data (HDF5 and NetCDF4)? We have plenty of binary file readers that aren't these file formats and handle some strange file structures so I'm sure we could make it work.

From what I know, the older GOES satellites also have NetCDF data files. Is there a reason these can't be used instead?

[djhoese]

FYI I brought this up to some coworkers and they had a couple existing modules for reading gvar in python that I didn't know about. Things like https://gitlab.ssec.wisc.edu/cspp_geo/gvar/blob/master/gvar/area/mcarea/area.py

[jferencik]

@djhoese I was aware SSEC has some native python code for McIDAS Area and some hybrid (C/Fortran/Python) for GVAR but the reader I have is a bit different. For start, its only dep is numpy.

The GVAR reader first does some inspection until it finds a vallid block0 from where it extracts info on how much mem to allocate, version (GVAR comes in three versions), the channels that are present, etc. It also prepares metadat for the user (resolutions of the channels, scan times for each line, and other stuff that will help the second phase , reading the block payload(data)

Then it maps the whole content into memory and starts parsing scan swath after scan swath. It does a lot of CRC and parity checks (to make sure the data and headers are valid) and this slows it down a bit.

What would be required to integrate this reader into pysat?. Does it fit the conceptual framework of pysat readers?

[djhoese]

Satpy (not pysat) only requires that data end up in an xarray DataArray object at the end of the day. The data is stored in a dask array which can be created from a numpy array. That means:

DataArray(dask_array, dims=('y', 'x'), attrs=metadata_dict)

Where metadata_dict includes things like area which is a pyresample AreaDefinition, standard_name, platform_name, sensor, units, etc. This is all described (kind of) https://satpy.readthedocs.io/en/latest/dev_guide/custom_reader.html

We don't need to worry too much about performance unless it is unusable.

[jferencik]

I had a brief look at the docs, and I believe it is doable. I would however need to get familiar with satpy first (specially configs). I will publish pygvar on github as well, and, I do not want to promise anything but will try to port/integrate pyarea and pygvar into satpy.

[mraspaud]

That would be great! I'll assign you to the issue then :)

[jhbravo]

hello @jferencik some weeks ago I was work with data downloaded from adde-server through McIdas-V, first I downloaded to NetCDF and managed to create a reader with satpy and apparently it works well However, as a pop-up window appears when writing the data, it disables me for a few seconds the window in which I work, that is annoying and more if I pretend to do it periodically I decided to download the data in area fromat and with PIL I managed to read the data and metadata similar to the one obtained in the NetCDF, I hope to resume this reader soon and I can share it with you

[sofiaermida]

Hi! I was wondering if there was any development regarding this issue. I need to read some old GEOS data in GVAR format and I can't find anything to read this file format with python. If you could help it would be really appreciated! Thanks

[mraspaud]

Not that I know of, unfortunately... Any help is appreciated!

[jferencik]

Hello Sofia,

https://github.com/jferencik/pygvar

this might help.

On Thu, 2 Feb 2023 at 13:48, Sofia Ermida @.***> wrote:

Hi! I was wondering if there was any development regarding this issue. I need to read some old GEOS data in GVAR format and I can't find anything to read this file format with python. If you could help it would be really appreciated! Thanks

— Reply to this email directly, view it on GitHub https://github.com/pytroll/satpy/issues/690#issuecomment-1413692652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7HXOPMZ6DV42SLNF76Z5LWVOULTANCNFSM4HDDV2RQ . You are receiving this because you were mentioned.Message ID: @.***>

[jferencik]

An example can be found here. https://github.com/jferencik/pygvar/blob/master/pygvar/test/read.py

be aware the reader will try to do its best when encountering damaged GVARs

On Thu, 2 Feb 2023 at 13:48, Sofia Ermida @.***> wrote:

Hi! I was wondering if there was any development regarding this issue. I need to read some old GEOS data in GVAR format and I can't find anything to read this file format with python. If you could help it would be really appreciated! Thanks

— Reply to this email directly, view it on GitHub https://github.com/pytroll/satpy/issues/690#issuecomment-1413692652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7HXOPMZ6DV42SLNF76Z5LWVOULTANCNFSM4HDDV2RQ . You are receiving this because you were mentioned.Message ID: @.***>

[mraspaud]

I'm closing this issue for now, as I believe @jferencik provided a solution. If someone wants to integrate this to satpy, feel free to open a PR or another issue for it.

[jferencik]

Hello Martin. I planned actually to do it but I am not working anymore with SolarGIS and in my current position I am not able to contribute I am sorry for this.

On Tue, 7 Feb 2023 at 08:33, Martin Raspaud @.***> wrote:

I'm closing this issue for now, as I believe @jferencik https://github.com/jferencik provided a solution. If someone wants to integrate this to satpy, feel free to open a PR or another issue for it.

— Reply to this email directly, view it on GitHub https://github.com/pytroll/satpy/issues/690#issuecomment-1420316191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7HXOKCMCCFSSOKBP6SPADWWH3E5ANCNFSM4HDDV2RQ . You are receiving this because you were mentioned.Message ID: @.***>

[mraspaud]

No problem, thank you for your contribution so far and putting out pygvar for free!