search

Unidata / LDM

The Unidata Local Data Manager (LDM) system includes network client and server programs designed for event-driven data distribution, and is the fundamental component of the Unidata Internet Data Distribution (IDD) system.
http://www.unidata.ucar.edu/software/ldm
Other
43 stars 27 forks source link

pqact option of -gunzip #99

Closed sebenste closed 2 years ago

sebenste commented 2 years ago

More and more, to save bandwidth, the NWS is sending files across their fiber or NOAAport feeds gzipped. I would like an -gunzip and an -unzip option in pqact to automatically unzip the file before writing to disk or running an action on the server. This would save a big extra step in post-processing of these files.

NaderCHASER commented 2 years ago

An "-unzip" flag that covers both GZipped and Zipped files would be nice to have.

I think that Unidata's MRMS feed comes in GZipped and is re-inserted decompressed so this might even help with that use case.

semmerson commented 2 years ago

This can be done by piping the data-product (using the "-close" option) to a script that decompresses its standard input on the fly into a file and then uses the AWIPS script "/awips2/ldm/dev called notifyAWIPS2-unidata.py" to notify Qpid about the file.