Failure to read nexrad file getting "unpack requires a string argument of length 46"

louiswicker commented 5 years ago

All,

I have several examples of the py-art nexrad reader failing with level-2 files - these are delivered real time via LDM.

For an example, data from MS/AL, at 2155 today, I cannot read this without throwing an error. I am using python2.7 from Anaconda (I know I need to switch!), PyART 1.9.2 (but 1.8 yields the same result). This looks like the MSG5 error that was reported last Aug, but seems to have not been squashed.

Thanks!

Lou Wicker, NSSL

radar file: KGWX_20190418_2155

Error message:

File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_archive.py", line 112, in read_nexrad_archive nfile = NEXRADLevel2File(prepare_for_read(filename)) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 160, in init pos, dic = _get_record_from_buf(buf, pos) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 587, in _get_record_from_buf new_pos = _get_msg5_from_buf(buf, pos, dic) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 703, in _get_msg5_from_buf dic['cut_parameters'].append(_unpack_from_buf(buf, pos2, MSG_5_ELEV)) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 715, in _unpack_from_buf return _unpack_structure(buf[pos:pos + size], structure) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 721, in _unpack_structure lst = struct.unpack(fmt, string) struct.error: unpack requires a string argument of length 46

scollis commented 5 years ago

Thanks @louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?

@rcjackson can you try this out please.

louiswicker commented 5 years ago

Scott,

its from conda on a linux64 box.

1.9.2

I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)

Lou

On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:

Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?

@rcjackson https://github.com/rcjackson can you try this out please.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316

I "Yet all is not lost," Francis said. "Human beings, while	capable of the worst, are also capable of rising above	themselves, choosing again what is good, and making	a new start, despite their mental and social conditioning."
Pope Francis

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

scollis commented 5 years ago

Cool. Bobby will check the latest source code to see if we get a failure.. We have a new release coming any day now and that might just fix this out of the box.

From: louiswicker notifications@github.com Reply-To: ARM-DOE/pyart reply@reply.github.com Date: Thursday, April 18, 2019 at 6:10 PM To: ARM-DOE/pyart pyart@noreply.github.com Cc: Scott scollis.acrf@gmail.com, Comment comment@noreply.github.com Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)

Scott,

its from conda on a linux64 box.

1.9.2

I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)

Lou

On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:

Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?

@rcjackson https://github.com/rcjackson can you try this out please.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316

I "Yet all is not lost," Francis said. "Human beings, while	capable of the worst, are also capable of rising above	themselves, choosing again what is good, and making	a new start, despite their mental and social conditioning."
Pope Francis

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

louiswicker commented 5 years ago

That would be great. We are operating for real starting April 29, so if I need to pull source to fix this and test next week let me know

Thanks for the fast responses

Lou

On Thu, Apr 18, 2019 at 6:12 PM Scott notifications@github.com wrote:

Cool. Bobby will check the latest source code to see if we get a failure.. We have a new release coming any day now and that might just fix this out of the box.

From: louiswicker notifications@github.com Reply-To: ARM-DOE/pyart reply@reply.github.com Date: Thursday, April 18, 2019 at 6:10 PM To: ARM-DOE/pyart pyart@noreply.github.com Cc: Scott scollis.acrf@gmail.com, Comment comment@noreply.github.com Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)

Scott,

its from conda on a linux64 box.

1.9.2

I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)

Lou

On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:

Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?

@rcjackson https://github.com/rcjackson can you try this out please.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ .

Dr. Louis J. Wicker NSSL/FRDD Rm 3336 National Weather Center 120 David L. Boren Boulevard, Norman, OK 73072 https://maps.google.com/?q=120+David+L.+Boren+Boulevard,+Norman,+OK+73072&entry=gmail&source=g

E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov

HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker

Phone: (405) 325-6340

Fax: (405) 325-2316

I "Yet all is not lost," Francis said. "Human beings, while capable of the worst, are also capable of rising above themselves, choosing again what is good, and making a new start, despite their mental and social conditioning."

Pope Francis

"The contents of this message are mine personally and do not reflect any position of the Government or NOAA."

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484719198, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPQC67GUSJHYODQG3U3PRD555ANCNFSM4HHA24WQ .

--

Dr. Louis J. Wicker NSSL/FRDD Rm 4368 National Weather Center 120 David L. Boren Boulevard, Norman, OK 73072

E-mail: Louis.Wicker@noaa.gov

HTTP: www.nssl.noaa.gov/~lwicker

Phone: (405) 325-6340

Fax: (405) 325-6780

"...In fact the hardest part of research is always to

find a question that's big enough thats worth answering

but little enough that you actually can answer it."

Professor Ed Witten (2012)

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072 https://maps.google.com/?q=120+David+L.+Boren+Boulevard,+Norman,+OK+73072&entry=gmail&source=g
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316

I "Yet all is not lost," Francis said. "Human beings, while	capable of the worst, are also capable of rising above	themselves, choosing again what is good, and making	a new start, despite their mental and social conditioning."
Pope Francis

Dr. Louis J. Wicker	NSSL/FRDD Rm 4368	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-6780
"...In fact the hardest part of research is always to
find a question that's big enough thats worth answering
but little enough that you actually can answer it."
Professor Ed Witten (2012)

|

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

scollis commented 5 years ago

@zssherman ping..

Even a micro release may help @louiswicker

rcjackson commented 5 years ago

When reading that time from the data on the NEXRAD Level 2 Archive, I cannot reproduce the error. This is with a python 3.7 environment and the latest conda release of Py-ART.

scollis commented 5 years ago

@rcjackson could you please make an env that installs Py-ART from the Conda Package?

scollis commented 5 years ago

Then we can see if the issue is either the data on S3 is different (ie an LDM issue) or if it is, indeed, that a release will fix things

rcjackson commented 5 years ago

It works in this environment with the latest conda release. It could be an LDM issue.

name: pyart_env channels:

default
conda-forge dependencies:
arm_pyart
python=3.7
boto3

scollis commented 5 years ago

@louiswicker is there a way to save out files that cause this error? We are going to need to look at if they are different to what ends up going to Amazon which is what we are using to try to replicate this issues.. Ping @dopplershift .. Any reason why a file collected off LDM would be different to what went to S3?

louiswicker commented 5 years ago

I have the file, and have put it on:

https://www.dropbox.com/s/szlnioisy5zv4sp/KGWX_20190418_2155?dl=0 https://www.dropbox.com/s/szlnioisy5zv4sp/KGWX_20190418_2155?dl=0

Let me know.

I can build a 3.7 env this morning.

Lou

On Apr 19, 2019, at 8:24 AM, Scott notifications@github.com wrote:

@louiswicker https://github.com/louiswicker is there a way to save out files that cause this error? We are going to need to look at if they are different to what ends up going to Amazon which is what we are using to try to replicate this issues.. Ping @dopplershift https://github.com/dopplershift .. Any reason why a file collected off LDM would be different to what went to S3?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484895117, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPQZCVLORCEZ2H5IVSTPRHBYFANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316
Tell me and I forget. Teach me and I remember.
Involve me and I learn.
- Benjamin Franklin

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

louiswicker commented 5 years ago

my install is from conda

On Apr 19, 2019, at 8:04 AM, Scott notifications@github.com wrote:

@rcjackson https://github.com/rcjackson could you please make an env that installs Py-ART from the Conda Package?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484889651, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPUIAO6FQF7OWGKG4HLPRG7N7ANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316
Tell me and I forget. Teach me and I remember.
Involve me and I learn.
- Benjamin Franklin

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

scollis commented 5 years ago

Great.. @rcjackson can you test with that file.. much appreciated all

louiswicker commented 5 years ago

All,

I pulled down the KGWX20190418_215557_V06 file from Amazon, and it does appear that PyART reads the file…so this may be an LDM issue on our end.

I would appreciate if someone could simply try to read that file I put on Dropbox, and if that fails, if you can let me know then I know its the LDM server.

Thanks for being so helpful - beers are owed!

Lou

On Apr 19, 2019, at 9:16 AM, Scott notifications@github.com wrote:

Great.. @rcjackson https://github.com/rcjackson can you test with that file.. much appreciated all

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPX4GKRHDBLJ4DRN2BDPRHH5BANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316

I "Yet all is not lost," Francis said. "Human beings, while	capable of the worst, are also capable of rising above	themselves, choosing again what is good, and making	a new start, despite their mental and social conditioning."
Pope Francis

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

rcjackson commented 5 years ago

I got the error when using it on that file, so there is a difference between the files downloaded via LDM and those on S3 that is causing the error.

Bobby

From: Scott notifications@github.com Sent: Friday, April 19, 2019 9:16:48 AM To: ARM-DOE/pyart Cc: Jackson, Robert; Mention Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)

Great.. @rcjacksonhttps://github.com/rcjackson can you test with that file.. much appreciated all

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5E6RBLRRCPJLNRRZN3PRHH5BANCNFSM4HHA24WQ.

louiswicker commented 5 years ago

Bobby,

thanks for the checking this out!

Lou

On Apr 19, 2019, at 9:50 AM, Bobby Jackson notifications@github.com wrote:

I got the error when using it on that file, so there is a difference between the files downloaded via LDM and those on S3 that is causing the error.

Bobby

From: Scott notifications@github.com Sent: Friday, April 19, 2019 9:16:48 AM To: ARM-DOE/pyart Cc: Jackson, Robert; Mention Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)

Great.. @rcjacksonhttps://github.com/rcjackson can you test with that file.. much appreciated all

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5E6RBLRRCPJLNRRZN3PRHH5BANCNFSM4HHA24WQ. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484919238, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPVDNBGDQUZJWKMNSDDPRHL2PANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316
Tell me and I forget. Teach me and I remember.
Involve me and I learn.
- Benjamin Franklin

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

dopplershift commented 5 years ago

@scollis I can open the file without error with MetPy's Level2File, but I do get a warning about Unknown message: 29. This is the message type that corresponds to the model data (type M) messages. I'm not currently saving those messages to the S3 archive, so that's why the S3 data work fine.

My advice would be to update your parser to at least skip past the unknown message types--the header is standard and contains enough information to skip to the next message. We're not yet parsing the type 29 messages in MetPy, though it's on the list Unidata/MetPy#95.

louiswicker commented 5 years ago

Thanks Ryan!

Lou

On Apr 19, 2019, at 2:11 PM, Ryan May notifications@github.com wrote:

@scollis https://github.com/scollis I can open the file without error with MetPy's Level2File, but I do get a warning about Unknown message: 29. This is the message type that corresponds to the model data (type M) messages. I'm not currently saving those messages to the S3 archive, so that's why the S3 data work fine.

My advice would be to update your parser to at least skip past the unknown message types--the header is standard and contains enough information to skip to the next message. We're not yet parsing the type 29 messages in MetPy, though it's on the list Unidata/MetPy#95 https://github.com/Unidata/MetPy/issues/95.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484989471, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPXMOOU6F2ILHBRORB3PRIKMXANCNFSM4HHA24WQ.

Dr. Louis J. Wicker	NSSL/FRDD Rm 3336	National Weather Center	120 David L. Boren Boulevard, Norman, OK 73072
E-mail: Louis.Wicker@noaa.gov
HTTP: www.nssl.noaa.gov/~lwicker
Phone: (405) 325-6340
Fax: (405) 325-2316
Tell me and I forget. Teach me and I remember.
Involve me and I learn.
- Benjamin Franklin

	"The contents of this message are mine personally and	do not reflect any position of the Government or NOAA."

scollis commented 5 years ago

Perfect. I am heading to China on Monday, I am concerned I will not get to this. @rcjackson can you put together a PR that ignores unknown messages? It would be great if they throw a warning like MetPy does

rcjackson commented 5 years ago

I've narrowed it down to an incomplete message 5. While Py-ART is expecting 48 bytes for the buffer, there are only 10 bytes available in the file. The message is at the very end of the file, so there are no more bytes to read.

dopplershift commented 5 years ago

@rcjackson Are you sure about that message type 5 at the end of the file? I don't show that in the file @louiswicker provided.

This is what I show with debugging output at the end of reading:

Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56213, date=18005, time_ms=79369655, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56214, date=18005, time_ms=79369665, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56215, date=18005, time_ms=79369715, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56216, date=18005, time_ms=79369763, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=65535, rda_channel=['Redundant Channel 1', 'ORDA'], msg_type=29, seq_num=0, date=0, time_ms=0, num_segments=12, segment_num=42490) Total message size: 828934 Unknown message: 29

dopplershift commented 5 years ago

Are you all aware that with the message 29 changes, they added special behavior for when the message header size (in half-words) has a value of 65535? In that case you're supposed to take the number of segments together with the segment number and treat them as a 32-bit int. In metpy we do:

if msg_hdr.size_hw == 65535:
    msg_bytes = msg_hdr.num_segments << 16 | msg_hdr.segment_num

See the 2620002R ICD for more information.

rcjackson commented 5 years ago

We were not aware of that. I just put a check for that into nexrad_level2.py and it seems to fix the problem.

scollis commented 5 years ago

Greetings from Nanjing all!

@rcjackson I assume this made it into Impressionism? If so, @louiswicker can you update Py-ART and let us know if this helps.

rcjackson commented 5 years ago

I think so, since it was merged before @zssherman did his release.

scollis commented 4 years ago

Closing this as @rcjackson 's update should have fixed this.. @louiswicker or @dopplershift please reopen if I am mistaken

melovato commented 1 year ago

Hello,

I am getting the same "error: unpack requires a buffer of 48 bytes" when reading CSU-CHILL CHL data. I have tried both pyart.io.read(filename) and pyart.io.ready_chl(filename) and the same error occurs. I am using Python 3.9 and arm_pyart 1.14.1 that I installed via conda. I can provide the package list and any other information if needed.

Thank you, Mateo

dopplershift commented 1 year ago

@melovato You should open a new issue since this one is regarding a problem with reading NEXRAD data, and CSU-CHILL data use a different format and reader.

ARM-DOE / pyart

Failure to read nexrad file getting "unpack requires a string argument of length 46" #826

--

|