Nexrad Level 3 unsupported product code

[josephhardinee]

When reading in a lvl 3 tar file from KTFG received the following error:

NotImplementedError: Level3 product with code 18259 is not supported

Is it possible to have this be issued as a warning instead of an error and just step over unsupported products, or load them in unnormalized similar to how the iris files are handled?

[josephhardinee]

I guess the second issue would be...does this actually support loading directly from the tar file?

[jjhelmus]

The NEXRAD Level 3 reader does not do support loading directly from a tar file, nor do I think it should. Trying to support inline unpacking/decompression of various archive formats get complicated fast and can be handled easily outside the read function. I'll add support for reading from a file-like object so that it will be possible to open the file with the standard library tarfile module and pass a member to the read_nexrad_level3 function. You can already do the same with most (all?) of the other file types supported by Py-ART.

NEXRAD Level 2 supports reading directly from bzip2 files as some service was serving files in that format (NCDC) and it was simple enough to add and the bz2 module was already being imported. In hindsight it probably would have been better to refer users to the bz2 module and have them pass a BZ2File to the read_nexrad_level2 to be more inline with other read functions.

Finally it should be mentioned that pyart.io.read will attempt to perform inline decompression of gzip and bzip2 files. This cannot be done for archive files (tar, zip) as they can contain multiple files. Passing a the file-like handler will work for these using the tarfile or* zipfile* modules.

[jjhelmus]

The more tricky question is can the NEXRAD Level 3 reader perform partial reading of unsupported products. The Level 3 or NIDS format is a message based file format which can contain radial, Cartesian, and some other more obscure layouts. Only radial data can be mapped to the Radar data model, other formats cannot. Hence the NEXRADLevel3 class has only implemented support for reading radial Level 3 data products and raises the NotImplementedError for non-radial data products. I'm not sure how these could be skipped over exactly, there is no data to map into the Radar class so what would the expected return be, an empty Radar object? I think raising the NotImplementedError is more agreeable, you can always catch this error if you are iterating over files.

[jjhelmus]

PR #312 Adds support for reading NEXRAD Level 3 files from a file like object. The following snippet can be used to read from tarfiles:

import pyart
import tarfile
tf = tarfile.TarFile('foo.tar')
f = tf.extractfile(tf.getmember('KLOT_SDUS33_N1PLOT_201304170004'))
radar = pyart.io.read_nexrad_level3(f)

@josephhardinee Let me know if you still are interested in handling the NotImplemented issue differently. If not I'll close this issue once the PR is merged.

[josephhardinee]

Hmm okay let me think on this a little more. I didn't realize lvl3 files were available in radial format(Most my work is with research networks). In particular for the grid files however, I assume they are all on the same grid(per lvl), and so would it be possible to set something up to accumulate them all into a single radar object? Otherwise for each volume you end up with 300 or so radar objects(per moment). There is the add_field functionality which I think would make this somewhat easy, but I guess I'm wondering if there would be interest in this type of workflow.

[jjhelmus]

Level 3 radial files contain a PPI scan from a single elevation scan for a particular moment or product (rain rate, etc). Multiple moments from the same radar and time period could be combined with the add_field method of the Radar class. Combining multiple sweeps is more tricky. When fixed/re-designed the join_radar function (PR #127) could be used to make Radar objects composable so that multiple level 3 files could be used to build up an entire radar volume from the individual sweeps and moments. Implemented Sweep and Ray classes (PR #242) could make process more user friendly.

This method requires a lot of work to generate a more complete but still not comprehensive NEXRAD volume. A Level 2 file provides the complete volume in a single file, I think directing users at these files when this level of information is needed is more prudent. Level 2 files do not provide KDP, rain rates and a few other higher order products which Level 3 file provide. Also the moments in Level 2 data are not corrected (velocity dealiasing, etc) where as the Level 3 data is so there still cases where Level 3 data provides helpful information.

[jjhelmus]

Changing label to Large Scale as implementing fixing the join_radar function is required to build full radar volumes from a series of Level 3 files. Reading Level 3 data from a file like object, for example a TarFile member, has been implemented which was the Easy Fix part of this issue.

[agumartina]

Hello @jjhelmus. I'm having a similar issue. When I try to open with pyart.io.read_nexrad_level3 a NEXRAD Level 3 file (Base reflectivity - KBOU_SDUS25_N1QFTG_201306181803), it return "NotImplementedError: Level3 product with code 0 is not supported". And if I export from NIDS to NetCDF with wct-viewer and open that NetCDF (v 3) file with pyart.io.read it returns "ValueError: max() arg is an empty sequence". Any help?

[jjhelmus]

@agumartina Can you provide a link to the file causing issues? Product N1Q is a base reflectivity product which should be supported by Py-ART, but KBOU does not seem to be a valid NEXRAD site and the FTG part of the filename suggests to me that this may be a Free Text message in NIDS format.

Converting to NetCDF using wct-viewer will not work with Py-ART, NetCDF files need to be in the CfRadial format which wct-viewer does not produce. That said performing a ncdump -h on that file would at least provide some insight into what is contained in the Level 3 file.

[zssherman]

Being looked into on the xradar side, closing.