Describe the bug
The Scene object is not able to load data from recently downloaded NUCAPS data from NOAA Class. There seems to have been a time format string change within the files. There is a format change of the time_coverage_start and time_coverage_end strings. I believe the format of the time variable is unchanged.
from datetime import datetime
from satpy import find_files_and_readers, Scene
from satpy.utils import debug_on; debug_on()
files = find_files_and_readers(start_time=datetime(2020,10,1,8,29), end_time=datetime(2020,10,1,8,32))
scn = Scene(files)
Expected behavior
The Scene object should load the files without error.
Actual results
[DEBUG: 2020-10-02 12:24:14 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/joleenf/code/satpy/satpy/etc'
[DEBUG: 2020-10-02 12:24:14 : satpy.readers.yaml_reader] Reading ('/Users/joleenf/code/satpy/satpy/etc/readers/nucaps.yaml',)
[DEBUG: 2020-10-02 12:24:14 : satpy.readers.yaml_reader] Assigning to nucaps: ['NUCAPS-EDR_v2r0_npp_s202010010831030_e202010010831330_c202010011004460.nc', 'NUCAPS-EDR_v2r0_npp_s202010010829270_e202010010829570_c202010011018340.nc', 'NUCAPS-EDR_v2r0_npp_s202010010831350_e202010010832050_c202010011018180.nc', 'NUCAPS-EDR_v2r0_npp_s202010010828550_e202010010829250_c202010011017490.nc', 'NUCAPS-EDR_v2r0_npp_s202010010830310_e202010010831010_c202010011003140.nc', 'NUCAPS-EDR_v2r0_npp_s202010010829590_e202010010830290_c202010011018170.nc']
Traceback (most recent call last):
File "", line 1, in
File "/Users/joleenf/code/satpy/satpy/scene.py", line 151, in init
self._readers = self._create_reader_instances(filenames=filenames,
File "/Users/joleenf/code/satpy/satpy/scene.py", line 200, in _create_reader_instances
return load_readers(filenames=filenames,
File "/Users/joleenf/code/satpy/satpy/readers/init.py", line 495, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 604, in create_filehandlers
filehandlers = self._new_filehandlers_for_filetype(filetype_info,
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 592, in _new_filehandlers_for_filetype
return list(filtered_iter)
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 561, in filter_fh_by_metadata
filehandler.metadata['start_time'] = filehandler.start_time
File "/Users/joleenf/code/satpy/satpy/readers/nucaps.py", line 90, in start_time
return self._parse_datetime(self['/attr/time_coverage_start'])
File "/Users/joleenf/code/satpy/satpy/readers/nucaps.py", line 84, in _parse_datetime
return datetime.strptime(datestr, "%Y-%m-%dT%H:%M:%S.%fZ")
File "/Users/joleenf/miniconda3/envs/satpy-dev/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/Users/joleenf/miniconda3/envs/satpy-dev/lib/python3.8/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '2020-10-01T08:31:03Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
Environment Info:
OS: OSX 10.15.6 (Darwin simple.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 x86_64)
Satpy Version: 0.23
PyResample Version: 1.16.0
Readers and writers dependencies Readers
abi_l1b: ok
abi_l1b_scmi: ok
abi_l2_nc: ok
acspo: ok
agri_l1: ok
ahi_hrit: ok
ahi_hsd: ok
ahi_l1b_gridded_bin: ok
ami_l1b: ok
amsr2_l1b: ok
amsr2_l2: ok
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
cmsaf-claas2_l2_nc: ok
electrol_hrit: ok
fci_l1c_fdhsi: ok
fci_l2_nc: ok
generic_image: ok
geocat: ok
ghrsst_l3c_sst: cannot find module 'satpy.readers.ghrsst_l3c_sst' (cannot import name 'Dataset' from 'satpy.dataset' (/Users/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/dataset/init.py))
glm_l2: ok
goes-imager_hrit: ok
goes-imager_nc: ok
gpm_imerg: ok
grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5: ok
iasi_l2: ok
iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
jami_hrit: ok
li_l2: cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf')
maia: ok
mersi2_l1b: ok
mimicTPW2_comp: ok
modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
mtsat2-imager_hrit: ok
nucaps: ok
nwcsaf-geo: ok
nwcsaf-msg2013-hdf5: ok
nwcsaf-pps_nc: ok
olci_l1b: ok
olci_l2: ok
omps_edr: ok
safe_sar_l2_ocn: ok
sar-c_safe: ok
scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/Users/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/dataset/init.py))
seviri_l1b_hrit: ok
seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native: ok
seviri_l1b_nc: ok
seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b: ok
slstr_l2: ok
smos_l2_wind: ok
tropomi_l2: ok
vaisala_gld360: ok
vii_l1b_nc: ok
vii_l2_nc: ok
viirs_compact: ok
viirs_edr_active_fires: ok
viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b: ok
viirs_sdr: ok
virr_l1b: ok
Writers
cf: ok
geotiff: ok
mitiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi: ok
simple_image: ok
Extras
cartopy: No module named 'cartopy'
geoviews: No module named 'geoviews'
Looks like maybe we should do a double check. First with the format with the .%f part and then if that fails, try again without the .%f part (note the . is in the format which makes it a required character).
Describe the bug The Scene object is not able to load data from recently downloaded NUCAPS data from NOAA Class. There seems to have been a time format string change within the files. There is a format change of the time_coverage_start and time_coverage_end strings. I believe the format of the time variable is unchanged.
To Reproduce Get NUCAPS EDR data from NOAA Class: (https://www.avl.class.noaa.gov/saa/products/search?datatype_family=JPSS_SND)
Expected behavior The Scene object should load the files without error.
Actual results [DEBUG: 2020-10-02 12:24:14 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/joleenf/code/satpy/satpy/etc' [DEBUG: 2020-10-02 12:24:14 : satpy.readers.yaml_reader] Reading ('/Users/joleenf/code/satpy/satpy/etc/readers/nucaps.yaml',) [DEBUG: 2020-10-02 12:24:14 : satpy.readers.yaml_reader] Assigning to nucaps: ['NUCAPS-EDR_v2r0_npp_s202010010831030_e202010010831330_c202010011004460.nc', 'NUCAPS-EDR_v2r0_npp_s202010010829270_e202010010829570_c202010011018340.nc', 'NUCAPS-EDR_v2r0_npp_s202010010831350_e202010010832050_c202010011018180.nc', 'NUCAPS-EDR_v2r0_npp_s202010010828550_e202010010829250_c202010011017490.nc', 'NUCAPS-EDR_v2r0_npp_s202010010830310_e202010010831010_c202010011003140.nc', 'NUCAPS-EDR_v2r0_npp_s202010010829590_e202010010830290_c202010011018170.nc'] Traceback (most recent call last): File "", line 1, in
File "/Users/joleenf/code/satpy/satpy/scene.py", line 151, in init
self._readers = self._create_reader_instances(filenames=filenames,
File "/Users/joleenf/code/satpy/satpy/scene.py", line 200, in _create_reader_instances
return load_readers(filenames=filenames,
File "/Users/joleenf/code/satpy/satpy/readers/init.py", line 495, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 604, in create_filehandlers
filehandlers = self._new_filehandlers_for_filetype(filetype_info,
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 592, in _new_filehandlers_for_filetype
return list(filtered_iter)
File "/Users/joleenf/code/satpy/satpy/readers/yaml_reader.py", line 561, in filter_fh_by_metadata
filehandler.metadata['start_time'] = filehandler.start_time
File "/Users/joleenf/code/satpy/satpy/readers/nucaps.py", line 90, in start_time
return self._parse_datetime(self['/attr/time_coverage_start'])
File "/Users/joleenf/code/satpy/satpy/readers/nucaps.py", line 84, in _parse_datetime
return datetime.strptime(datestr, "%Y-%m-%dT%H:%M:%S.%fZ")
File "/Users/joleenf/miniconda3/envs/satpy-dev/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/Users/joleenf/miniconda3/envs/satpy-dev/lib/python3.8/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '2020-10-01T08:31:03Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
Environment Info:
OS: OSX 10.15.6 (Darwin simple.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64 x86_64)
Satpy Version: 0.23
PyResample Version: 1.16.0
Readers and writers dependencies Readers
abi_l1b: ok abi_l1b_scmi: ok abi_l2_nc: ok acspo: ok agri_l1: ok ahi_hrit: ok ahi_hsd: ok ahi_l1b_gridded_bin: ok ami_l1b: ok amsr2_l1b: ok amsr2_l2: ok avhrr_l1b_aapp: ok avhrr_l1b_eps: ok avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac') avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac') caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf') clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf') cmsaf-claas2_l2_nc: ok electrol_hrit: ok fci_l1c_fdhsi: ok fci_l2_nc: ok generic_image: ok geocat: ok ghrsst_l3c_sst: cannot find module 'satpy.readers.ghrsst_l3c_sst' (cannot import name 'Dataset' from 'satpy.dataset' (/Users/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/dataset/init.py)) glm_l2: ok goes-imager_hrit: ok goes-imager_nc: ok gpm_imerg: ok grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib') hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib') hy2_scat_l2b_h5: ok iasi_l2: ok iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'"))) jami_hrit: ok li_l2: cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf') maia: ok mersi2_l1b: ok mimicTPW2_comp: ok modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf') modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf') msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur') mtsat2-imager_hrit: ok nucaps: ok nwcsaf-geo: ok nwcsaf-msg2013-hdf5: ok nwcsaf-pps_nc: ok olci_l1b: ok olci_l2: ok omps_edr: ok safe_sar_l2_ocn: ok sar-c_safe: ok scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/Users/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/dataset/init.py)) seviri_l1b_hrit: ok seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf') seviri_l1b_native: ok seviri_l1b_nc: ok seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) slstr_l1b: ok slstr_l2: ok smos_l2_wind: ok tropomi_l2: ok vaisala_gld360: ok vii_l1b_nc: ok vii_l2_nc: ok viirs_compact: ok viirs_edr_active_fires: ok viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf') viirs_l1b: ok viirs_sdr: ok virr_l1b: ok Writers
cf: ok geotiff: ok mitiff: ok ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff') scmi: ok simple_image: ok Extras
cartopy: No module named 'cartopy' geoviews: No module named 'geoviews'