Closed louiswicker closed 4 years ago
Thanks @louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?
@rcjackson can you try this out please.
Scott,
its from conda on a linux64 box.
1.9.2
I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)
Lou
On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:
Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?
@rcjackson https://github.com/rcjackson can you try this out please.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
I "Yet all is not lost," Francis said. "Human beings, while | capable of the worst, are also capable of rising above | themselves, choosing again what is good, and making | a new start, despite their mental and social conditioning." |
---|---|---|---|
Pope Francis |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
Cool. Bobby will check the latest source code to see if we get a failure.. We have a new release coming any day now and that might just fix this out of the box.
From: louiswicker notifications@github.com Reply-To: ARM-DOE/pyart reply@reply.github.com Date: Thursday, April 18, 2019 at 6:10 PM To: ARM-DOE/pyart pyart@noreply.github.com Cc: Scott scollis.acrf@gmail.com, Comment comment@noreply.github.com Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)
Scott,
its from conda on a linux64 box.
1.9.2
I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)
Lou
On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:
Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?
@rcjackson https://github.com/rcjackson can you try this out please.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
I "Yet all is not lost," Francis said. "Human beings, while | capable of the worst, are also capable of rising above | themselves, choosing again what is good, and making | a new start, despite their mental and social conditioning." |
---|---|---|---|
Pope Francis |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
That would be great. We are operating for real starting April 29, so if I need to pull source to fix this and test next week let me know
Thanks for the fast responses
Lou
On Thu, Apr 18, 2019 at 6:12 PM Scott notifications@github.com wrote:
Cool. Bobby will check the latest source code to see if we get a failure.. We have a new release coming any day now and that might just fix this out of the box.
From: louiswicker notifications@github.com Reply-To: ARM-DOE/pyart reply@reply.github.com Date: Thursday, April 18, 2019 at 6:10 PM To: ARM-DOE/pyart pyart@noreply.github.com Cc: Scott scollis.acrf@gmail.com, Comment comment@noreply.github.com Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)
Scott,
its from conda on a linux64 box.
1.9.2
I have also tried it in metPy, and it complains, but I can see some data. Also converting it using RadxConvert does not yield a cfradial-1 or UF file that pyart can read (long shot here..)
Lou
On Apr 18, 2019, at 5:54 PM, Scott notifications@github.com wrote:
Thanks @louiswicker https://github.com/louiswicker Is your install from source (GitHub) or conda install... what does pyart.version yeild?
@rcjackson https://github.com/rcjackson can you try this out please.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484715737>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AADUKPS7HCK42SZHGO3AAU3PRD3ZVANCNFSM4HHA24WQ .
Dr. Louis J. Wicker NSSL/FRDD Rm 3336 National Weather Center 120 David L. Boren Boulevard, Norman, OK 73072 https://maps.google.com/?q=120+David+L.+Boren+Boulevard,+Norman,+OK+73072&entry=gmail&source=g E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker Phone: (405) 325-6340 Fax: (405) 325-2316
I "Yet all is not lost," Francis said. "Human beings, while capable of the worst, are also capable of rising above themselves, choosing again what is good, and making a new start, despite their mental and social conditioning." Pope Francis
"The contents of this message are mine personally and do not reflect any position of the Government or NOAA."
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484719198, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPQC67GUSJHYODQG3U3PRD555ANCNFSM4HHA24WQ .
--
Dr. Louis J. Wicker NSSL/FRDD Rm 4368 National Weather Center 120 David L. Boren Boulevard, Norman, OK 73072 E-mail: Louis.Wicker@noaa.gov HTTP: www.nssl.noaa.gov/~lwicker Phone: (405) 325-6340 Fax: (405) 325-6780 "...In fact the hardest part of research is always to find a question that's big enough thats worth answering but little enough that you actually can answer it." Professor Ed Witten (2012)
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
@zssherman ping..
Even a micro release may help @louiswicker
When reading that time from the data on the NEXRAD Level 2 Archive, I cannot reproduce the error. This is with a python 3.7 environment and the latest conda release of Py-ART.
@rcjackson could you please make an env that installs Py-ART from the Conda Package?
Then we can see if the issue is either the data on S3 is different (ie an LDM issue) or if it is, indeed, that a release will fix things
It works in this environment with the latest conda release. It could be an LDM issue.
name: pyart_env channels:
@louiswicker is there a way to save out files that cause this error? We are going to need to look at if they are different to what ends up going to Amazon which is what we are using to try to replicate this issues.. Ping @dopplershift .. Any reason why a file collected off LDM would be different to what went to S3?
I have the file, and have put it on:
https://www.dropbox.com/s/szlnioisy5zv4sp/KGWX_20190418_2155?dl=0 https://www.dropbox.com/s/szlnioisy5zv4sp/KGWX_20190418_2155?dl=0
Let me know.
I can build a 3.7 env this morning.
Lou
On Apr 19, 2019, at 8:24 AM, Scott notifications@github.com wrote:
@louiswicker https://github.com/louiswicker is there a way to save out files that cause this error? We are going to need to look at if they are different to what ends up going to Amazon which is what we are using to try to replicate this issues.. Ping @dopplershift https://github.com/dopplershift .. Any reason why a file collected off LDM would be different to what went to S3?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484895117, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPQZCVLORCEZ2H5IVSTPRHBYFANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
Tell me and I forget. Teach me and I remember. | |||
Involve me and I learn. | |||
- Benjamin Franklin |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
my install is from conda
On Apr 19, 2019, at 8:04 AM, Scott notifications@github.com wrote:
@rcjackson https://github.com/rcjackson could you please make an env that installs Py-ART from the Conda Package?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484889651, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPUIAO6FQF7OWGKG4HLPRG7N7ANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
Tell me and I forget. Teach me and I remember. | |||
Involve me and I learn. | |||
- Benjamin Franklin |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
Great.. @rcjackson can you test with that file.. much appreciated all
All,
I pulled down the KGWX20190418_215557_V06 file from Amazon, and it does appear that PyART reads the file…so this may be an LDM issue on our end.
I would appreciate if someone could simply try to read that file I put on Dropbox, and if that fails, if you can let me know then I know its the LDM server.
Thanks for being so helpful - beers are owed!
Lou
On Apr 19, 2019, at 9:16 AM, Scott notifications@github.com wrote:
Great.. @rcjackson https://github.com/rcjackson can you test with that file.. much appreciated all
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPX4GKRHDBLJ4DRN2BDPRHH5BANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov mailto:Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker http://www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
I "Yet all is not lost," Francis said. "Human beings, while | capable of the worst, are also capable of rising above | themselves, choosing again what is good, and making | a new start, despite their mental and social conditioning." |
---|---|---|---|
Pope Francis |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
I got the error when using it on that file, so there is a difference between the files downloaded via LDM and those on S3 that is causing the error.
Bobby
From: Scott notifications@github.com Sent: Friday, April 19, 2019 9:16:48 AM To: ARM-DOE/pyart Cc: Jackson, Robert; Mention Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)
Great.. @rcjacksonhttps://github.com/rcjackson can you test with that file.. much appreciated all
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5E6RBLRRCPJLNRRZN3PRHH5BANCNFSM4HHA24WQ.
Bobby,
thanks for the checking this out!
Lou
On Apr 19, 2019, at 9:50 AM, Bobby Jackson notifications@github.com wrote:
I got the error when using it on that file, so there is a difference between the files downloaded via LDM and those on S3 that is causing the error.
Bobby
From: Scott notifications@github.com Sent: Friday, April 19, 2019 9:16:48 AM To: ARM-DOE/pyart Cc: Jackson, Robert; Mention Subject: Re: [ARM-DOE/pyart] Failure to read nexrad file getting "unpack requires a string argument of length 46" (#826)
Great.. @rcjacksonhttps://github.com/rcjackson can you test with that file.. much appreciated all
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ARM-DOE/pyart/issues/826#issuecomment-484908997, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5E6RBLRRCPJLNRRZN3PRHH5BANCNFSM4HHA24WQ. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484919238, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPVDNBGDQUZJWKMNSDDPRHL2PANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
Tell me and I forget. Teach me and I remember. | |||
Involve me and I learn. | |||
- Benjamin Franklin |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
@scollis I can open the file without error with MetPy's Level2File
, but I do get a warning about Unknown message: 29
. This is the message type that corresponds to the model data (type M) messages. I'm not currently saving those messages to the S3 archive, so that's why the S3 data work fine.
My advice would be to update your parser to at least skip past the unknown message types--the header is standard and contains enough information to skip to the next message. We're not yet parsing the type 29 messages in MetPy, though it's on the list Unidata/MetPy#95.
Thanks Ryan!
Lou
On Apr 19, 2019, at 2:11 PM, Ryan May notifications@github.com wrote:
@scollis https://github.com/scollis I can open the file without error with MetPy's Level2File, but I do get a warning about Unknown message: 29. This is the message type that corresponds to the model data (type M) messages. I'm not currently saving those messages to the S3 archive, so that's why the S3 data work fine.
My advice would be to update your parser to at least skip past the unknown message types--the header is standard and contains enough information to skip to the next message. We're not yet parsing the type 29 messages in MetPy, though it's on the list Unidata/MetPy#95 https://github.com/Unidata/MetPy/issues/95.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARM-DOE/pyart/issues/826#issuecomment-484989471, or mute the thread https://github.com/notifications/unsubscribe-auth/AADUKPXMOOU6F2ILHBRORB3PRIKMXANCNFSM4HHA24WQ.
Dr. Louis J. Wicker | NSSL/FRDD Rm 3336 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 |
---|---|---|---|
E-mail: Louis.Wicker@noaa.gov | |||
HTTP: www.nssl.noaa.gov/~lwicker | |||
Phone: (405) 325-6340 | |||
Fax: (405) 325-2316 | |||
Tell me and I forget. Teach me and I remember. | |||
Involve me and I learn. | |||
- Benjamin Franklin |
"The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." |
---|
Perfect. I am heading to China on Monday, I am concerned I will not get to this. @rcjackson can you put together a PR that ignores unknown messages? It would be great if they throw a warning like MetPy does
I've narrowed it down to an incomplete message 5. While Py-ART is expecting 48 bytes for the buffer, there are only 10 bytes available in the file. The message is at the very end of the file, so there are no more bytes to read.
@rcjackson Are you sure about that message type 5 at the end of the file? I don't show that in the file @louiswicker provided.
This is what I show with debugging output at the end of reading:
Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56213, date=18005, time_ms=79369655, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56214, date=18005, time_ms=79369665, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56215, date=18005, time_ms=79369715, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=1008, rda_channel='ORDA', msg_type=31, seq_num=56216, date=18005, time_ms=79369763, num_segments=1, segment_num=1) Total message size: 2028 Got message: MsgHdr(size_hw=65535, rda_channel=['Redundant Channel 1', 'ORDA'], msg_type=29, seq_num=0, date=0, time_ms=0, num_segments=12, segment_num=42490) Total message size: 828934 Unknown message: 29
Are you all aware that with the message 29 changes, they added special behavior for when the message header size (in half-words) has a value of 65535? In that case you're supposed to take the number of segments together with the segment number and treat them as a 32-bit int. In metpy we do:
if msg_hdr.size_hw == 65535:
msg_bytes = msg_hdr.num_segments << 16 | msg_hdr.segment_num
See the 2620002R ICD for more information.
We were not aware of that. I just put a check for that into nexrad_level2.py and it seems to fix the problem.
Greetings from Nanjing all!
@rcjackson I assume this made it into Impressionism? If so, @louiswicker can you update Py-ART and let us know if this helps.
I think so, since it was merged before @zssherman did his release.
Closing this as @rcjackson 's update should have fixed this.. @louiswicker or @dopplershift please reopen if I am mistaken
Hello,
I am getting the same "error: unpack requires a buffer of 48 bytes" when reading CSU-CHILL CHL data. I have tried both pyart.io.read(filename) and pyart.io.ready_chl(filename) and the same error occurs. I am using Python 3.9 and arm_pyart 1.14.1 that I installed via conda. I can provide the package list and any other information if needed.
Thank you, Mateo
@melovato You should open a new issue since this one is regarding a problem with reading NEXRAD data, and CSU-CHILL data use a different format and reader.
All,
I have several examples of the py-art nexrad reader failing with level-2 files - these are delivered real time via LDM.
For an example, data from MS/AL, at 2155 today, I cannot read this without throwing an error. I am using python2.7 from Anaconda (I know I need to switch!), PyART 1.9.2 (but 1.8 yields the same result). This looks like the MSG5 error that was reported last Aug, but seems to have not been squashed.
Thanks!
Lou Wicker, NSSL
radar file: KGWX_20190418_2155
Error message:
File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_archive.py", line 112, in read_nexrad_archive nfile = NEXRADLevel2File(prepare_for_read(filename)) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 160, in init pos, dic = _get_record_from_buf(buf, pos) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 587, in _get_record_from_buf new_pos = _get_msg5_from_buf(buf, pos, dic) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 703, in _get_msg5_from_buf dic['cut_parameters'].append(_unpack_from_buf(buf, pos2, MSG_5_ELEV)) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 715, in _unpack_from_buf return _unpack_structure(buf[pos:pos + size], structure) File "/home/louis.wicker/anaconda2/envs/wof-test/lib/python2.7/site-packages/pyart/io/nexrad_level2.py", line 721, in _unpack_structure lst = struct.unpack(fmt, string) struct.error: unpack requires a string argument of length 46