search

ARM-DOE / pyart

The Python-ARM Radar Toolkit. A data model driven interactive toolkit for working with weather radar data.
https://arm-doe.github.io/pyart/
Other
516 stars 268 forks source link

pyart.io.read_cradial issue on a Mac (Big Sur 11.6.5) #1078

Closed wxwolff closed 2 years ago

wxwolff commented 2 years ago

Hi,

I recently encounter an issue reading Cfradial files on my Mac. It used to work, but now gets a MaskError. I can run on Linux with lots of warnings, but no errors.

Pyart: 1.11.8 Numpy: 1.21.2 Found 54 files to process. File Type: cf Unzipping CF file... Processing: NPOL1_2018_0320_075650.cf

MaskError Traceback (most recent call last) /var/folders/vc/wrsvlv556kx14jvkf03s6p9c0000gn/T/ipykernel_36460/3570699089.py in <cell line: 110>() 137 f = ungzip_file(cf) 138 print("Processing: ",f) --> 139 radar = pyart.io.read(f, file_field_names=True) 140 else: 141 f = cf

~/anaconda3/lib/python3.8/site-packages/pyart/io/auto_read.py in read(filename, use_rsl, kwargs) 110 else: 111 dset.close() --> 112 return read_cfradial(filename, kwargs) # CF/Radial 113 if filetype == 'WSR88D': 114 return read_nexrad_archive(filename, **kwargs)

~/anaconda3/lib/python3.8/site-packages/pyart/io/cfradial.py in read_cfradial(filename, field_names, additional_metadata, file_field_names, exclude_fields, include_fields, delay_field_loading, **kwargs) 118 # 4.3 Global variable -> move to metadata dictionary 119 if 'volume_number' in ncvars: --> 120 metadata['volume_number'] = int(ncvars['volume_number'][:]) 121 else: 122 metadata['volume_number'] = 0

~/anaconda3/lib/python3.8/site-packages/numpy/ma/core.py in int(self) 4382 "to Python scalars") 4383 elif self._mask: -> 4384 raise MaskError('Cannot convert masked element to a Python int.') 4385 return int(self.item()) 4386

MaskError: Cannot convert masked element to a Python int.

zssherman commented 2 years ago

@wxwolff Seems the metadata is storing a mask value instead of possibly a string. I think I might have a work around for that. Do you have a file I can test on?

wxwolff commented 2 years ago

Hi Zach,

Thanks for the really quick response. I do have a file, but it’s too big for email so I have posted it on our web site. Try this link:

https://gpm-gv.gsfc.nasa.gov/Data/NPOL1_2022_0102_151855.cf.gz

Dave

David B. Wolff, Chief Code 610.W GPM Deputy Project Scientist for GV NASA Wallops Flight Facility 34200 Fulton Street Building N159, Room E220 Wallops Island, VA 23337 Ph: 757-824-1515; Cell: 443-262-1146

From: Zach Sherman @.> Reply-To: ARM-DOE/pyart @.> Date: Friday, March 18, 2022 at 10:43 AM To: ARM-DOE/pyart @.> Cc: "Wolff, David B. (WFF-610W)" @.>, Mention @.***> Subject: [EXTERNAL] Re: [ARM-DOE/pyart] pyart.io.read_cradial issue on a Mac (Big Sur 11.6.5) (Issue #1078)

@wxwolffhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwxwolff&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Ceb46b86d2c684d79537808da08eda6cd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832114057415260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mJzDl1UjiUrTHxCW4pgBn2cn5WYskfVsN%2BZ87LwQPkI%3D&reserved=0 Seems the metadata is storing a mask value instead of possibly a string. I think I might have a work around for that. Do you have a file I can test on?

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FARM-DOE%2Fpyart%2Fissues%2F1078%23issuecomment-1072478134&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Ceb46b86d2c684d79537808da08eda6cd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832114057415260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=i8MsPfrE%2Bl3z35SNU9QkkVuKK1UB7EqyfrjGLFghEWY%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADJRUYXRJB4MMF23LNOMYLTVASJAPANCNFSM5RB75KJQ&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Ceb46b86d2c684d79537808da08eda6cd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832114057415260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AUg3uddIAvPtp5MEHaBPyFd2NyMFLUkfVXC5WPYyZaM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Ceb46b86d2c684d79537808da08eda6cd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832114057415260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8c0kIe6dl4JZbExJIUOyAZi%2BFt4wVSN%2F6MO6vUTvwyA%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Ceb46b86d2c684d79537808da08eda6cd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832114057415260%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1mQmDivMOmhd%2FPO3idgCOGuiXcsxBEjUIlbbqe1twVo%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

zssherman commented 2 years ago

@wxwolff Not a problem! And I haven't been able to reproduce the error yet. Are you adding metadata to the readers? Are you able to share the code your using that is causing this error?

zssherman commented 2 years ago

Ah you said mac is not working but linux is however. Gotcha so its probably os related, ignore my last comment, I think I have a fix that will work for mac

wxwolff commented 2 years ago

Yes, it is a mac issue, but from what I can tell, at its root, it’s a numpy issue.

Davids-Mac-Pro.local> python --version Python 3.8.12 Davids-Mac-Pro.local> python Python 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:33:09) [Clang 11.1.0 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import pyart

You are using the Python ARM Radar Toolkit (Py-ART), an open source

library for working with weather radar data. Py-ART is partly

supported by the U.S. Department of Energy as part of the Atmospheric

Radiation Measurement (ARM) Climate Research Facility, an Office of

Science user facility.

If you use this software to prepare a publication, please cite:

JJ Helmus and SM Collis, JORS 2016, doi: 10.5334/jors.119

f = 'NPOL1_2018_0320_075650.cf' radar = pyart.io.read_cfradial(f) Traceback (most recent call last): File "", line 1, in File "/Users/wolff/anaconda3/lib/python3.8/site-packages/pyart/io/cfradial.py", line 120, in read_cfradial metadata['volume_number'] = int(ncvars['volume_number'][:]) File "/Users/wolff/anaconda3/lib/python3.8/site-packages/numpy/ma/core.py", line 4384, in int raise MaskError('Cannot convert masked element to a Python int.') numpy.ma.core.MaskError: Cannot convert masked element to a Python int.

David B. Wolff, Chief Code 610.W GPM Deputy Project Scientist for GV NASA Wallops Flight Facility 34200 Fulton Street Building N159, Room E220 Wallops Island, VA 23337 Ph: 757-824-1515; Cell: 443-262-1146

From: Zach Sherman @.> Reply-To: ARM-DOE/pyart @.> Date: Friday, March 18, 2022 at 11:08 AM To: ARM-DOE/pyart @.> Cc: "Wolff, David B. (WFF-610W)" @.>, Mention @.***> Subject: [EXTERNAL] Re: [ARM-DOE/pyart] pyart.io.read_cradial issue on a Mac (Big Sur 11.6.5) (Issue #1078)

@wxwolffhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwxwolff&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cc954d9920ca9482cd89908da08f124fb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832129045429191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2an0usLUdsUD26xu%2F%2BHupwxmEJ%2FIHGzSxA%2FJB9pAhm0%3D&reserved=0 Not a problem! And I haven't been able to reproduce the error yet. Are you adding metadata to the readers? Are you able to share the code your using that is causing this error?

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FARM-DOE%2Fpyart%2Fissues%2F1078%23issuecomment-1072501636&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cc954d9920ca9482cd89908da08f124fb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832129045429191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2BBo8t0UNQUEzni5IKAOpe9CUjmqr5M2mEORJbdBQqoM%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADJRUYVTVYAWORYZTGHQN3DVASL6HANCNFSM5RB75KJQ&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cc954d9920ca9482cd89908da08f124fb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832129045429191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=58IwWDsRNFnJZJ2WT3N6J7p1qmbuOUUYQWVRF3trm9M%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cc954d9920ca9482cd89908da08f124fb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832129045429191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MPEEqlRxqMRtPbdhKwDyxZ1VrvPTS43qcs14oR25%2BQQ%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cc954d9920ca9482cd89908da08f124fb%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832129045429191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojjdl%2Ba%2F4eFG%2BvaqCZ8OWj0AfQwXwnShlU0L20izV5o%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

zssherman commented 2 years ago

@wxwolff Yeah, I'm confused on why it works for linux but not mac. I do have a possible fix which Ill be doing a PR here shortly. Are you locally installing pyart or is it conda installed?

wxwolff commented 2 years ago

Conda.

David B. Wolff, Chief Code 610.W GPM Deputy Project Scientist for GV NASA Wallops Flight Facility 34200 Fulton Street Building N159, Room E220 Wallops Island, VA 23337 Ph: 757-824-1515; Cell: 443-262-1146

From: Zach Sherman @.> Reply-To: ARM-DOE/pyart @.> Date: Friday, March 18, 2022 at 11:24 AM To: ARM-DOE/pyart @.> Cc: "Wolff, David B. (WFF-610W)" @.>, Mention @.***> Subject: [EXTERNAL] Re: [ARM-DOE/pyart] pyart.io.read_cradial issue on a Mac (Big Sur 11.6.5) (Issue #1078)

@wxwolffhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwxwolff&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cd8ddab5bcef84acd375408da08f36c0d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832138823159733%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Wup7p3pzphq%2Ft0z6UriD8O8pVjUGtGBXwv3T%2FuFLcck%3D&reserved=0 Yeah, I'm confused on why it works for linux but not mac. I do have a possible fix which Ill be doing a PR here shortly. Are you locally installing pyart or is it conda installed?

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FARM-DOE%2Fpyart%2Fissues%2F1078%23issuecomment-1072517454&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cd8ddab5bcef84acd375408da08f36c0d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832138823159733%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rKPSXNiatGNjWNidZhyzr%2BSub9Gj4Nl30KFv4jd%2BK8w%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADJRUYQPR5ILWMDMPTSAJA3VASN3NANCNFSM5RB75KJQ&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cd8ddab5bcef84acd375408da08f36c0d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832138823315961%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2BjEQeeSowEBWTAlX%2BIdjeFM1e5p2TlWyTgSBj7iMeoY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cd8ddab5bcef84acd375408da08f36c0d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832138823315961%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=faek3waFO8E%2Bu9tyaxD1ZR35%2BpL3zv3z%2Ff%2FP%2BSbQv3M%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CDavid.B.Wolff%40nasa.gov%7Cd8ddab5bcef84acd375408da08f36c0d%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637832138823315961%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sd8amcoXgwhAO8q%2BMqPs94Vit7O2KkbbP0x0d3XxlW4%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

zssherman commented 2 years ago

@wxwolff Trying to find a way to test this real quick, sadly i'm on a linux system. But in theory the PR #1080 should fix it as it will flatten and retrieve the volume number before using int on it.

mgrover1 commented 2 years ago

@wxwolff which package/function is this? What are you using for unzipping in python?

137 f = ungzip_file(cf)
mgrover1 commented 2 years ago

After unzipping the file from the command line, I was able to read in the file with the following setup:

wxwolff commented 2 years ago

1080 resolved my issue! Thanks Zach!!