search

ARM-DOE / pyart

The Python-ARM Radar Toolkit. A data model driven interactive toolkit for working with weather radar data.
https://arm-doe.github.io/pyart/
Other
513 stars 266 forks source link

Compressed data ended before the end-of-stream marker was reached #1186

Closed wxbinge closed 2 years ago

wxbinge commented 2 years ago

This error is easily resolved by redownloading the radar data, but I'm running this particular script operationally, so when this error occurs, the script exits and I would miss the cycle.

I was wondering is there a way to bypass this error?

I am pulling the radar data from the TGFTP server.

Traceback (most recent call last): File "/wfs/radar/pyscripts/national_composite_archival.py", line 186, in radar.append(pyart.io.read_nexrad_level3(i)) # Opening radar file and appending each file to an array File "/anaconda3/envs/WFS/lib/python3.10/site-packages/pyart/io/nexradl3_read.py", line 78, in read_nexrad_level3 nfile = NEXRADLevel3File(prepare_for_read(filename, storage_options=storage_options)) File "/anaconda3/envs/WFS/lib/python3.10/site-packages/pyart/io/nexrad_level3.py", line 146, in init buf2 = bz2.decompress(buf[bpos:]) File "/anaconda3/envs/WFS/lib/python3.10/bz2.py", line 341, in decompress raise ValueError("Compressed data ended before the " ValueError: Compressed data ended before the end-of-stream marker was reached

zssherman commented 2 years ago

@wxbinge Hmm would maybe a try and except, catch the error and rerun the download on the file work? or are you using pyarts nexrad reader to download the file?

wxbinge commented 2 years ago

I don't know why 'try and except' didn't cross my mind, but you're right. That will work to bypass the error. I'm using bash to download the radar files, so I'm going to need to do something creative to see if I can invoke a download from within the python script if the error arises.

mgrover1 commented 2 years ago

@wxbinge that error occurs when the file is not completely downloaded.

See the stack overflow discussion here

You will need to add logic within your script to check to see if the file is completely downloaded yet, or consider streaming the data from Amazon Web Services (AWS), as we did in this example: