Closed arthur-e closed 1 year ago
Hi,
Thanks for raising this. It looks like you unzipped the files into a folder named SLSTR
. Satpy requires files to be unzipped into a folder with the same name (without extension) as the zip file itself, so in this case the folder name should be:
S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004
Could you try placing the files into a folder with that name and seeing if it works? If this fixes it, I'll add an additional comment in the notebook specifying how to unzip the data. Let me know how you get on!
Hi @simonrp84, I made the change to the folder name:
$ ls /home/arthur/Downloads/PyCoxMunk/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/
cartesian_an.nc flags_an.nc geometry_tn.nc S2_radiance_an.nc S5_quality_bo.nc S8_BT_in.nc
cartesian_ao.nc flags_ao.nc geometry_to.nc S2_radiance_ao.nc S5_radiance_an.nc S8_BT_io.nc
cartesian_bn.nc flags_bn.nc indices_an.nc S3_quality_an.nc S5_radiance_ao.nc S8_quality_in.nc
cartesian_bo.nc flags_bo.nc indices_ao.nc S3_quality_ao.nc S5_radiance_bn.nc S8_quality_io.nc
cartesian_fn.nc flags_fn.nc indices_bn.nc S3_radiance_an.nc S5_radiance_bo.nc S9_BT_in.nc
cartesian_fo.nc flags_fo.nc indices_bo.nc S3_radiance_ao.nc S6_quality_an.nc S9_BT_io.nc
cartesian_in.nc flags_in.nc indices_fn.nc S4_quality_an.nc S6_quality_ao.nc S9_quality_in.nc
cartesian_io.nc flags_io.nc indices_fo.nc S4_quality_ao.nc S6_quality_bn.nc S9_quality_io.nc
cartesian_tx.nc geodetic_an.nc indices_in.nc S4_quality_bn.nc S6_quality_bo.nc time_an.nc
F1_BT_fn.nc geodetic_ao.nc indices_io.nc S4_quality_bo.nc S6_radiance_an.nc time_bn.nc
F1_BT_fo.nc geodetic_bn.nc met_tx.nc S4_radiance_an.nc S6_radiance_ao.nc time_in.nc
F1_quality_fn.nc geodetic_bo.nc S1_quality_an.nc S4_radiance_ao.nc S6_radiance_bn.nc viscal.nc
F1_quality_fo.nc geodetic_fn.nc S1_quality_ao.nc S4_radiance_bn.nc S6_radiance_bo.nc xfdumanifest.xml
F2_BT_in.nc geodetic_fo.nc S1_radiance_an.nc S4_radiance_bo.nc S7_BT_in.nc
F2_BT_io.nc geodetic_in.nc S1_radiance_ao.nc S5_quality_an.nc S7_BT_io.nc
F2_quality_in.nc geodetic_io.nc S2_quality_an.nc S5_quality_ao.nc S7_quality_in.nc
F2_quality_io.nc geodetic_tx.nc S2_quality_ao.nc S5_quality_bn.nc S7_quality_io.nc
However, I'm getting the same error as before.
I set a trace with ipdb
in the corresponding satpy
code, in the configs_for_reader()
loop (ca. L476).
473 for reader_configs in configs_for_reader(reader):
474 (reader_instance, loadables, this_sensor_supported) = _get_loadables_for_reader_config(
475 base_dir, reader, sensor, reader_configs, reader_kwargs, fs)
476 import ipdb
477 ipdb.set_trace()#FIXME
I'm seeing:
>>> list(configs_for_reader(reader))
[['/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml']]
I looked through this YAML file but I don't really understand it.
>>> reader_instance
<satpy.readers.yaml_reader.FileYAMLReader object at 0x7f4d28427160>
>>> loadables
set()
>>> this_sensor_supported
False
And in the call to _get_loadables_for_reader_config()
:
>>> base_dir
'/home/arthur/Downloads/PyCoxMunk/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004'
>>> reader
'slstr_l1b'
>>> sensor is None
True
>>> reader_configs
['/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml']
>>> reader_kwargs
{'filter_parameters': {}}
>>> fs is None
True
I'm reading this satpy
issue as well and I can also write:
And:
>>> from satpy.utils import check_satpy; check_satpy()
Readers
=======
abi_l1b: ok
abi_l1b_scmi: ok
abi_l2_nc: ok
acspo: cannot find module 'satpy.readers.acspo' (No module named 'netCDF4')
agri_fy4a_l1: cannot find module 'satpy.readers.agri_l1' (No module named 'h5py')
agri_fy4b_l1: cannot find module 'satpy.readers.agri_l1' (No module named 'h5py')
ahi_hrit: ok
ahi_hsd: ok
ahi_l1b_gridded_bin: ok
ami_l1b: cannot find module 'satpy.readers.ami_l1b' (No module named 'pyspectral')
amsr2_l1b: cannot find module 'satpy.readers.amsr2_l1b' (No module named 'h5py')
amsr2_l2: cannot find module 'satpy.readers.amsr2_l2' (No module named 'h5py')
amsr2_l2_gaasp: ok
amsub_l1c_aapp: cannot find 'FrequencyDoubleSideBand' in the module 'satpy.readers.aapp_mhs_amsub_l1c'
ascat_l2_soilmoisture_bufr: cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
atms_l1b_nc: cannot find module 'satpy.readers.atms_l1b_nc' (No module named 'netCDF4')
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'geotiepoints')
avhrr_l1c_eum_gac_fdr_nc: ok
caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'netCDF4')
cmsaf-claas2_l2_nc: cannot find module 'satpy.readers.cmsaf_claas2' (No module named 'netCDF4')
electrol_hrit: ok
fci_l1c_nc: cannot find module 'satpy.readers.fci_l1c_nc' (No module named 'netCDF4')
fci_l2_nc: ok
generic_image: cannot find module 'satpy.readers.generic_image' (No module named 'rasterio')
geocat: cannot find module 'satpy.readers.geocat' (No module named 'netCDF4')
ghi_l1: cannot find module 'satpy.readers.ghi_l1' (No module named 'h5py')
ghrsst_l2: ok
glm_l2: ok
goes-imager_hrit: ok
goes-imager_nc: ok
gpm_imerg: cannot find module 'satpy.readers.gpm_imerg' (No module named 'h5py')
grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5: cannot find module 'satpy.readers.hy2_scat_l2b_h5' (No module named 'h5py')
iasi_l2: cannot find module 'satpy.readers.iasi_l2' (No module named 'h5py')
iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
ici_l1b_nc: cannot find module 'satpy.readers.ici_l1b_nc' (No module named 'geotiepoints')
insat3d_img_l1b_h5: cannot find module 'satpy.readers.insat3d_img_l1b_h5' (No module named 'datatree' It can be installed with the xarray-datatree package.)
jami_hrit: ok
li_l2_nc: cannot find module 'satpy.readers.li_l2_nc' (No module named 'netCDF4')
maia: cannot find module 'satpy.readers.maia' (No module named 'h5py')
meris_nc_sen3: ok
mersi2_l1b: cannot find module 'satpy.readers.mersi2_l1b' (No module named 'pyspectral')
mhs_l1c_aapp: ok
mimicTPW2_comp: cannot find module 'satpy.readers.mimic_TPW2_nc' (No module named 'netCDF4')
mirs: ok
modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'rioxarray')
msu_gsa_l1b: cannot find module 'satpy.readers.msu_gsa_l1b' (No module named 'h5py')
mtsat2-imager_hrit: ok
mviri_l1b_fiduceo_nc: ok
mwi_l1b_nc: cannot find module 'satpy.readers.ici_l1b_nc' (No module named 'geotiepoints')
mws_l1b_nc: cannot find module 'satpy.readers.mws_l1b' (No module named 'netCDF4')
nucaps: cannot find module 'satpy.readers.nucaps' (No module named 'netCDF4')
nwcsaf-geo: ok
nwcsaf-msg2013-hdf5: cannot find module 'satpy.readers.nwcsaf_msg2013_hdf5' (No module named 'h5py')
nwcsaf-pps_nc: ok
oceancolorcci_l3_nc: cannot find module 'satpy.readers.oceancolorcci_l3_nc' (No module named 'netCDF4')
olci_l1b: ok
olci_l2: ok
omps_edr: cannot find module 'satpy.readers.omps_edr' (No module named 'h5py')
safe_sar_l2_ocn: ok
sar-c_safe: cannot find module 'satpy.readers.sar_c_safe' (No module named 'rasterio')
satpy_cf_nc: ok
scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (No module named 'h5py')
seadas_l2: cannot find module 'satpy.readers.seadas_l2' (No module named 'pyhdf')
seviri_l1b_hrit: ok
seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native: ok
seviri_l1b_nc: ok
seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b: ok
slstr_l2: cannot find module 'satpy.readers.slstr_l2' (No module named 'satpy.readers.slstr_l2')
smos_l2_wind: cannot find module 'satpy.readers.smos_l2_wind' (No module named 'netCDF4')
tropomi_l2: cannot find module 'satpy.readers.tropomi_l2' (No module named 'netCDF4')
vaisala_gld360: ok
vii_l1b_nc: cannot find module 'satpy.readers.vii_l1b_nc' (No module named 'geotiepoints')
vii_l2_nc: cannot find module 'satpy.readers.vii_l2_nc' (No module named 'geotiepoints')
viirs_compact: cannot find module 'satpy.readers.viirs_compact' (No module named 'h5py')
viirs_edr_active_fires: cannot find module 'satpy.readers.viirs_edr_active_fires' (No module named 'netCDF4')
viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b: cannot find module 'satpy.readers.viirs_l1b' (No module named 'netCDF4')
viirs_sdr: cannot find module 'satpy.readers.viirs_sdr' (No module named 'h5py')
virr_l1b: cannot find module 'satpy.readers.virr_l1b' (No module named 'pyspectral')
Writers
=======
awips_tiled: ok
cf: cannot find module 'satpy.writers.cf_writer' (Ensure that the netCDF4 or h5netcdf package is installed.)
geotiff: cannot find module 'satpy.writers.geotiff' (No module named 'rasterio')
mitiff: ok
ninjogeotiff: cannot find module 'satpy.writers.ninjogeotiff' (No module named 'rasterio')
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image: ok
Extras
======
cartopy: No module named 'cartopy'
geoviews: No module named 'geoviews'
But it seems that: slstr_l1b: ok
.
Following that same satpy
issue, I tried instead:
import glob
from satpy import Scene, find_files_and_readers
from pycoxmunk import PyCoxMunk
import numpy as np
from satpy.utils import debug_on
debug_on()
import warnings
warnings.filterwarnings('ignore')
# Set which bands we wish to load
bnames = ['S1', 'S2', 'S3']
# Set the directory containing the SLSTR data dir
# This folder should contain one or more subfolders with the SLSTR data (ending .SEN3)
# To save memory and CPU time, only use a few granules maximum for this.
indir = '/home/arthur/Downloads/PyCoxMunk/inputs'
# Set the directory where we'll save the output
outdir = '/home/arthur/Downloads/PyCoxMunk/'
files = glob.glob(f'{indir}/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/*')
And I have a new error! Well, a variation on the older error. Progress.
>>> scn = Scene(filenames = files, reader = "slstr_l1b")
[DEBUG: 2023-01-25 08:00:49 : satpy.readers.yaml_reader] Reading ('/usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
[WARNING: 2023-01-25 08:00:49 : satpy.readers.yaml_reader] No filenames found for reader: slstr_l1b
[WARNING: 2023-01-25 08:00:49 : satpy.readers] Don't know how to open the following files: {'/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/met_tx.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/S5_quality_bn.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004/F2_BT_io.nc',
...
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[6], line 1
----> 1 scn = Scene(filenames = files, reader = "slstr_l1b")
File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/scene.py:133, in Scene.__init__(self, filenames, reader, filter_parameters, reader_kwargs)
130 if filenames:
131 filenames = convert_remote_files_to_fsspec(filenames, storage_options)
--> 133 self._readers = self._create_reader_instances(filenames=filenames,
134 reader=reader,
135 reader_kwargs=cleaned_reader_kwargs)
136 self._datasets = DatasetDict()
137 self._wishlist = set()
File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/scene.py:154, in Scene._create_reader_instances(self, filenames, reader, reader_kwargs)
149 def _create_reader_instances(self,
150 filenames=None,
151 reader=None,
152 reader_kwargs=None):
153 """Find readers and return their instances."""
--> 154 return load_readers(filenames=filenames,
155 reader=reader,
156 reader_kwargs=reader_kwargs)
File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/readers/__init__.py:579, in load_readers(filenames, reader, reader_kwargs)
576 break
578 _check_remaining_files(remaining_filenames)
--> 579 _check_reader_instances(reader_instances)
580 return reader_instances
File /usr/local/python-env/PyCoxMunk/lib/python3.8/site-packages/satpy/readers/__init__.py:618, in _check_reader_instances(reader_instances)
616 def _check_reader_instances(reader_instances):
617 if not reader_instances:
--> 618 raise ValueError("No supported files found")
619 if not any(list(r.available_dataset_ids) for r in reader_instances.values()):
620 raise ValueError("No dataset could be loaded. Either missing "
621 "requirements (such as Epilog, Prolog) or none of the "
622 "provided files match the filter parameters.")
ValueError: No supported files found
So, it seems that despite slstr_l1b: ok
in the output of check_satpy()
, that reader is not really "ok"?
Also, I've since installed netCDF4
(and h5py
) in this virtual environment, since it seemed that it would be impossible to open *.nc
files otherwise. That was prior to getting the ValueError
above, however.
Are these supposed to be *.SEN3
files, instead, as a comment in Simple_SLSTR.ipynb
implies?
Hmm, this is curious. I'll download those files myself and see what problems I run into...
Ok, I tried this myself and both find_files_and_readers
and glob.glob
work for setting up the Scene.
However, I'm using a conda-based install, whereas you've been using pip. The satpy developers suggest that there may be a conflict as you're in the pycoxmunk
directory but not installing the files from that directory and that this may result in different / conflicting libraries being used.
Could you please re-try the installation you give in your initial message, but replace pip install pycoxmunk
with pip install .
?
pip install .
did not work. I don't usually use conda
because it is so slow, but I just installed miniconda to try it out...
conda activate
conda create -n PyCoxMunk
conda activate PyCoxMunk
conda install -c conda-forge pycoxmunk
Then, running the code from before, I'm getting the same error. I also get the error when running Simple_SLSTR.ipynb
after conda install notebook
. Here's the output from conda list
:
$ conda list
# packages in environment at /usr/local/dev/miniconda3/envs/PyCoxMunk:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
affine 2.4.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
asciitree 0.3.3 py_2 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
blosc 1.21.3 hafa529b_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boost-cpp 1.78.0 h75c5d50_1 conda-forge
brotlipy 0.7.0 py310h5764c6d_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
cfitsio 4.2.0 hd9d235c_0 conda-forge
cftime 1.6.2 py310hde88566_1 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
configobj 5.0.8 pyhd8ed1ab_0 conda-forge
cryptography 39.0.0 py310h34c0648_0 conda-forge
curl 7.87.0 hdc1c0ab_0 conda-forge
cytoolz 0.12.0 py310h5764c6d_1 conda-forge
dask 2023.1.0 pyhd8ed1ab_0 conda-forge
dask-core 2023.1.0 pyhd8ed1ab_0 conda-forge
distributed 2023.1.0 pyhd8ed1ab_0 conda-forge
docutils 0.19 py310hff52083_1 conda-forge
donfig 0.7.0 pyhd8ed1ab_1 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
expat 2.5.0 h27087fc_0 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.1 hc2a2eb6_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge
geos 3.11.1 h27087fc_0 conda-forge
geotiff 1.7.1 h7157cca_5 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
h5py 3.8.0 nompi_py310h0311031_100 conda-forge
hdf4 4.2.15 h9772cbc_5 conda-forge
hdf5 1.12.2 nompi_h4df4325_101 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jpeg 9e h166bdaf_2 conda-forge
json-c 0.16 hc379101_0 conda-forge
kealib 1.5.0 ha7026e8_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lcms2 2.14 hfd0df8a_1 conda-forge
ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcurl 7.87.0 hdc1c0ab_0 conda-forge
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgdal 3.6.2 h10cbb15_3 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h606061b_1 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libjpeg-turbo 2.1.4 h166bdaf_0 conda-forge
libkml 1.3.0 h37653c0_1015 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_5 conda-forge
libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge
libnghttp2 1.51.0 hff17c54_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.1 hb675445_3 conda-forge
librttopo 1.1.0 ha49c73b_12 conda-forge
libspatialite 5.0.1 h7c8129e_22 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.4 h166bdaf_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.10.3 h7463322_0 conda-forge
libzip 1.9.2 hc929e4a_1 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
llvmlite 0.39.1 py310h58363a5_1 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.2.0 py310h0cfdcf0_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.1.2 py310h1fa729e_0 conda-forge
msgpack-python 1.0.4 py310hbf28c38_1 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
netcdf4 1.6.2 nompi_py310h55e1e36_100 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.82 he02c5a1_0 conda-forge
numba 0.56.4 py310ha5257ce_0 conda-forge
numcodecs 0.11.0 py310heca2aa9_1 conda-forge
numpy 1.23.5 py310h53a5b5f_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.0.7 h0b41bf4_1 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 py310h9b08913_0 conda-forge
partd 1.3.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pillow 9.4.0 py310h4927cde_0 conda-forge
pip 22.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
poppler 22.12.0 h091648b_1 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 15.1 h3248436_3 conda-forge
proj 9.1.0 h8ffa02c_1 conda-forge
psutil 5.9.4 py310h5764c6d_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pycoxmunk 1.0.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pykdtree 1.3.6 py310hde88566_2 conda-forge
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyorbital 1.7.3 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.4.1 py310hfc24d34_0 conda-forge
pyresample 1.26.0.post0 py310h769672d_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pyspectral 0.12.3 pyhd8ed1ab_0 conda-forge
python 3.10.8 h4a9ceb5_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-geotiepoints 1.5.1 py310h0a54255_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2022.7.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py310h5764c6d_5 conda-forge
rasterio 1.3.4 py310hfc14bbd_0 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
satpy 0.39.0 pyhd8ed1ab_0 conda-forge
scipy 1.10.0 py310h8deb116_0 conda-forge
setuptools 66.1.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.9 hbd366e4_2 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlite 3.40.0 h4ff8645_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tiledb 2.13.2 hd532e3d_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.2 py310h5764c6d_1 conda-forge
tqdm 4.64.1 pyhd8ed1ab_0 conda-forge
trollimage 1.20.0 pyhd8ed1ab_0 conda-forge
trollsift 0.5.0 pyhd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzcode 2022g h166bdaf_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
urllib3 1.26.14 pyhd8ed1ab_0 conda-forge
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xarray 2023.1.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.4 h55805fa_1 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zarr 2.13.3 pyhd8ed1ab_0 conda-forge
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
hi @arthur-e , try the following
from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()
if __name__ == '__main__':
files = find_files_and_readers(base_dir="/home/**/slstr/", reader='slstr_l1b')
scn = Scene(filenames=files, reader='slstr_l1b')
my_channel = 'S1'
scn.load([my_channel])
#scn[my_channel].data
import matplotlib.pyplot as plt
plt.figure()
scn[my_channel].plot.imshow(cmap='viridis')
plt.show()
@arthur-e @simonrp84 i guess the problem is with solar zenith angles. it is because it is in a different stripe (tie point grid). but the commands above by default use the stripe<1> i.e 'a'
Hi @kameshvinjamuri, I saved your Python script as test.py
and ran it:
$ python test.py
/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
for entry_point in entry_points().get(name, []):
[DEBUG: 2023-01-25 13:16:32 : satpy.readers.yaml_reader] Reading ('/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
Traceback (most recent call last):
File "/usr/local/dev/PyCoxMunk/test.py", line 6, in <module>
files = find_files_and_readers(base_dir="/home/**/slstr/", reader='slstr_l1b')
File "/usr/local/dev/miniconda3/envs/PyCoxMunk/lib/python3.10/site-packages/satpy/readers/__init__.py", line 484, in find_files_and_readers
raise ValueError("No supported files found")
ValueError: No supported files found
I gather than Sentinel-3 has multiple overpasses ("stripes"?). Is there a way to specify which "stripe" to use? I am looking at the satpy
Quickstart and I don't see a relevant example.
@arthur-e can you repeat test.py with an exclusive satpy environment using conda ( i mean conda create new env, conda install satpy).
@arthur-e slstr have different stripe for different bands. most of the 500m bands are available in all stripes namely a, b, c, and tie point. for example, some of the fire channels at 1000 m resolution have only b/c stripe. a tie point stripe is something that is sampled for every 16 points (unfortunately solar zenith angles are only at tie point). we should interpolate to the resolution chosen. I suggest the slstr level 1 ATBD. (things mentioned here might change in the future with the new data product versions)
@kameshvinjamuri I have tried it in a fresh conda
environment:
conda create -c conda-forge -n satpy python satpy
conda activate satpy
import glob
from satpy import Scene, find_files_and_readers
import numpy as np
from satpy.utils import debug_on
debug_on()
import warnings
warnings.filterwarnings('ignore')
# Set which bands we wish to load
bnames = ['S1', 'S2', 'S3']
# Set the directory containing the SLSTR data dir
# This folder should contain one or more subfolders with the SLSTR data (ending .SEN3)
# To save memory and CPU time, only use a few granules maximum for this.
indir = '/home/arthur/Downloads/PyCoxMunk/inputs'
# Set the directory where we'll save the output
outdir = '/home/arthur/Downloads/PyCoxMunk/'
# Re-downloaded this granule using direct URL and it now has ".SEN3" appended to the folder name
files = glob.glob(f'{indir}/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/*')
scn = Scene(filenames = files, reader = "slstr_l1b")
And this is the output I obtained:
[DEBUG: 2023-01-25 14:50:52 : satpy.readers.yaml_reader] Reading ('/usr/local/dev/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
[DEBUG: 2023-01-25 14:50:52 : satpy.readers.yaml_reader] Assigning to slstr_l1b: ['/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S1_radiance_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S6_radiance_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S2_radiance_ao.nc',
...
[WARNING: 2023-01-25 14:50:54 : satpy.readers] Don't know how to open the following files: {'/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S6_quality_bn.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/indices_ao.nc', '/home/arthur/Downloads/PyCoxMunk/inputs/S3B_SL_1_RBT____20230123T191306_20230123T191505_20230123T202259_0119_075_198_5940_PS2_O_NR_004.SEN3/S2_quality_ao.nc',
...
That looks like it was successful, and that the scene was created.
The stripes etc should not be important in this case, by the way, and you don't need to specify them when creating or loading the Scene.
@simonrp84 I think I now understand what the issue is. In Examples/Simple_SLSTR.ipynb
there is written:
scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'))
However, what ultimately worked when I created a new environment was:
files = glob.glob(f'{indir}/*/*')
scn = Scene(filenames = files, reader = "slstr_l1b")
So, it seems that Scene
also needs to know what reader to use. In Examples/Simple_SLSTR.ipynb
, this would amount to:
scn = Scene(find_files_and_readers(base_dir=indir, reader='slstr_l1b'), reader = 'slstr_l1b')
Or, for clarity:
file_list = find_files_and_readers(base_dir = indir, reader = 'slstr_l1b')
scn = Scene(file_list, reader = 'slstr_l1b')
So, I created a new virtualenv
and installed PyCoxMunk in editable mode using pip
(working with my preferred installation approach for now):**
cd PyCoxMunk
pip install -e .
I found that netCDF4
and h5py
were not installed as part of the PyCoxMunk
installation:
ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
https://docs.xarray.dev/en/stable/user-guide/io.html
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
So I also added a note to the Notebook about needing to install netCDF4
for the example.
I put all this in a Pull Request. See PR #4. Glad we got this figured out! If PyCoxMunk users have no prior experience with satpy
, I think these changes will be essential to avoid confusion.
How I installed
PyCoxMunk
:What I did:
I installed Jupyter Notebook:
And launched it, navigating to the
Examples
folder. I openedSimple_SLSTR.ipynb
and ran through to the third cell:I am getting the following error and traceback:
And yet, I have the following files in the
indir
:Which are the data extracted from this ZIP archive downloaded from the Copernicus hub (URL):
I couldn't find
satpy
documentation onfind_files_and_readers()
but the source code gives me the impression I should be able to find a list of the expected files somewhere...My Python environment: