kameshvinjamuri commented 1 year ago

Describe the bug Hi, i am using satpy to read slstr l1b data....using the following code, and got the error

To Reproduce

# Your code here

from satpy import Scene

scn = Scene(filenames=['/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_radiance_an.nc'], reader='slstr_l1b')
scn.load(['S1_an']

Expected behavior read as xarray

Actual results

  File "/tmp/ipykernel_767887/2469185162.py", line 1, in <module>
    scn = Scene(filenames=['/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_radiance_an.nc'], reader='slstr_l1b')

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/scene.py", line 133, in __init__
    self._readers = self._create_reader_instances(filenames=filenames,

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/scene.py", line 154, in _create_reader_instances
    return load_readers(filenames=filenames,

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/__init__.py", line 570, in load_readers
    reader_instance.create_filehandlers(

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 612, in create_filehandlers
    filehandlers = self._new_filehandlers_for_filetype(filetype_info,

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 600, in _new_filehandlers_for_filetype
    return list(filtered_iter)

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 568, in filter_fh_by_metadata
    for filehandler in filehandlers:

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 509, in _new_filehandler_instances
    yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/slstr_l1b.py", line 137, in __init__
    self.stripe = filename_info['stripe']

KeyError: 'stripe'

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

OS: [Linux]
Satpy Version: [0.39.0]
PyResample Version:
Readers and writers dependencies (when relevant): [run from satpy.utils import check_satpy; check_satpy()]

Additional context is this due to the lack of filename_info, and file_type_info options?? What's the best way to deal slstr_l1b into an array or any georeferenced image? any suggestions are highly appreciated

pnuu commented 1 year ago

Hi!

Try passing the directory instead of the single .ncfile:

scn = Scene(filenames=['/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/'], reader='slstr_l1b')

My guess is the format is also using other files from the directory.

kameshvinjamuri commented 1 year ago

hi, tried but no luck. i tried cheking yaml files too, with/without 'frames'

scn = Scene(filenames=['/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/'], reader='slstr_l1b')
WARNING:satpy.readers.yaml_reader:No filenames found for reader: slstr_l1b
WARNING:satpy.readers:Don't know how to open the following files: {'/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/'}
Traceback (most recent call last):

  File "/tmp/ipykernel_767887/453325046.py", line 1, in <module>
    scn = Scene(filenames=['/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/'], reader='slstr_l1b')

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/scene.py", line 133, in __init__
    self._readers = self._create_reader_instances(filenames=filenames,

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/scene.py", line 154, in _create_reader_instances
    return load_readers(filenames=filenames,

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/__init__.py", line 579, in load_readers
    _check_reader_instances(reader_instances)

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/__init__.py", line 618, in _check_reader_instances
    raise ValueError("No supported files found")

ValueError: No supported files found

mraspaud commented 1 year ago

I usually use the find_files_and_readers function like this:

    files = find_files_and_readers(
                                   base_dir="/home/a001673/data/satellite/Sentinel-3/",
                                   reader='slstr_l2')
    scn = Scene(filenames=files, ...)

kameshvinjamuri commented 1 year ago

I usually use the find_files_and_readers function like this:

    files = find_files_and_readers(
                                   base_dir="/home/a001673/data/satellite/Sentinel-3/",
                                   reader='slstr_l2')
    scn = Scene(filenames=files, ...)

this one is for l2, but im using l1b

mraspaud commented 1 year ago

sure, just use the correct reader in that call.

kameshvinjamuri commented 1 year ago

no luck

files = find_files_and_readers(base_dir="/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/",
                                   reader='slstr_l1b')
scn = Scene(filenames=files)
Traceback (most recent call last):

  File "/tmp/ipykernel_767887/3186657089.py", line 1, in <module>
    files = find_files_and_readers(base_dir="/home/k*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/",

  File "/home/*h/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/__init__.py", line 484, in find_files_and_readers
    raise ValueError("No supported files found")

ValueError: No supported files found

simonrp84 commented 1 year ago

Can you switch debugging on please?

from satpy.utils import debug_on
debug_on()

I vaguely recall that the slstr reader needs an additional library, but this only shows up in the debugging info and not in the error message.

kameshvinjamuri commented 1 year ago

hi, this resulted in

from satpy.utils import debug_on
debug_on()
from satpy import Scene

files = find_files_and_readers(base_dir="/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/",
                                   reader='slstr_l1b')
scn = Scene(filenames=files)

/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
DEBUG:satpy.readers.yaml_reader:Reading ('/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
[DEBUG: 2023-01-23 16:07:26 : satpy.readers.yaml_reader] Reading ('/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/etc/readers/slstr_l1b.yaml',)
Traceback (most recent call last):

  File "/tmp/ipykernel_767887/2543651335.py", line 6, in <module>
    files = find_files_and_readers(base_dir="/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/",

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/__init__.py", line 484, in find_files_and_readers
    raise ValueError("No supported files found")

ValueError: No supported files found

mraspaud commented 1 year ago

@kameshvinjamuri I just tested with this script and it works for me:

from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()

if __name__ == '__main__':
    files = find_files_and_readers(base_dir="/home/a001673/data/satellite/Sentinel-3/",
                                   reader='slstr_l1b')
    scn = Scene(filenames=files)

    composite = "overview"
    scn.load([composite])

    nscn = scn.resample('maspalomas')
    nscn.save_dataset(composite)

see the result here: overview

So, to go further, we need to answer a few questions:

Do you have a complete fileset for the slstr data ? (my SL1RBT has 98 files in it)
What version of satpy are you running (you wrote 3, but there is no satpy version 3)
Could you provide the result of from satpy.utils import check_satpy; check_satpy() as asked in the issue template?

kameshvinjamuri commented 1 year ago

hi, it is running into the same old error,

yeah I have the complete dataset (mine is 111 files with 110 .nc and 1 .xml)
sorry I change the satpy version in the query (v 0.39.0)
the check_satpy() results are the following (with missing pyhdf issues)
Readers

/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select. for entry_point in entry_points().get(name, []): abi_l1b: ok abi_l1b_scmi: ok abi_l2_nc: ok acspo: ok agri_fy4a_l1: ok agri_fy4b_l1: ok ahi_hrit: ok ahi_hsd: ok ahi_l1b_gridded_bin: ok ami_l1b: ok amsr2_l1b: ok amsr2_l2: ok amsr2_l2_gaasp: ok amsub_l1c_aapp: cannot find 'FrequencyDoubleSideBand' in the module 'satpy.readers.aapp_mhs_amsub_l1c' ascat_l2_soilmoisture_bufr: cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'"))) atms_l1b_nc: ok avhrr_l1b_aapp: ok avhrr_l1b_eps: ok avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac') avhrr_l1b_hrpt: ok avhrr_l1c_eum_gac_fdr_nc: ok caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf') clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf') cmsaf-claas2_l2_nc: ok electrol_hrit: ok fci_l1c_nc: ok fci_l2_nc: ok generic_image: ok geocat: ok ghi_l1: ok ghrsst_l2: ok glm_l2: ok goes-imager_hrit: ok goes-imager_nc: ok gpm_imerg: ok grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib') hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib') hy2_scat_l2b_h5: ok iasi_l2: ok iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'"))) ici_l1b_nc: ok insat3d_img_l1b_h5: cannot find module 'satpy.readers.insat3d_img_l1b_h5' (No module named 'datatree' It can be installed with the xarray-datatree package.) jami_hrit: ok li_l2_nc: ok maia: ok meris_nc_sen3: ok mersi2_l1b: ok mhs_l1c_aapp: ok mimicTPW2_comp: ok mirs: ok modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf') modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf') msi_safe: ok msu_gsa_l1b: ok mtsat2-imager_hrit: ok mviri_l1b_fiduceo_nc: ok mwi_l1b_nc: ok mws_l1b_nc: ok nucaps: ok nwcsaf-geo: ok nwcsaf-msg2013-hdf5: ok nwcsaf-pps_nc: ok oceancolorcci_l3_nc: ok olci_l1b: ok olci_l2: ok omps_edr: ok safe_sar_l2_ocn: ok sar-c_safe: ok satpy_cf_nc: ok scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/home/kamesh/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/dataset/init.py)) seadas_l2: cannot find module 'satpy.readers.seadas_l2' (No module named 'pyhdf') seviri_l1b_hrit: ok seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf') seviri_l1b_native: ok seviri_l1b_nc: ok seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) slstr_l1b: ok slstr_l2: cannot find module 'satpy.readers.slstr_l2' (No module named 'satpy.readers.slstr_l2') smos_l2_wind: ok tropomi_l2: ok vaisala_gld360: ok vii_l1b_nc: ok vii_l2_nc: ok viirs_compact: ok viirs_edr_active_fires: ok viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf') viirs_l1b: ok viirs_sdr: ok virr_l1b: ok

Writers

awips_tiled: ok cf: ok geotiff: ok mitiff: ok ninjogeotiff: ok ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff') simple_image: ok

Extras

cartopy: No module named 'cartopy' geoviews: No module named 'geoviews'

mraspaud commented 1 year ago

Ok, nothing wrong with dependencies or versions then it seems. Could you share the data you are running on, so I can try locally?

kameshvinjamuri commented 1 year ago

okay, sure (let me know if the link is no accessible)

https://seafile.zfn.uni-bremen.de/d/80f4468df3a64de58a61/

mraspaud commented 1 year ago

@kameshvinjamuri it seems to work for me. I think you might need to change the base_dir parameter in your call. So, if I understand correctly, you have the data in /home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/, so the base_dir needs to be one step up, like this:

files = find_files_and_readers(base_dir="/home/*/Desktop/mpc/slstr/", reader='slstr_l1b')

Give that a shot :)

overview

mraspaud commented 1 year ago

An alternative is to do:

    files = glob.glob("/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/*")
    scn = Scene(filenames=files, reader="slstr_l1b")

kameshvinjamuri commented 1 year ago

showing this error now. can I know your python version?

  File "/home/*/anaconda3/envs/b01/lib/python3.10/site-packages/satpy/readers/slstr_l1b.py", line 137, in __init__
    self.stripe = filename_info['stripe']

KeyError: 'stripe'

mraspaud commented 1 year ago

python 3.10

kameshvinjamuri commented 1 year ago

okay, strange. one question: what exactly you plotted there? like the radiance of any band?

mraspaud commented 1 year ago

It's a colored composite. If you want to take a look at the first channel only for example, just do scn.load(["S1"])

kameshvinjamuri commented 1 year ago

yeah, that makes sense. but one more question is the slstr l1b specific, there are many stripes (grid) like a, b, i, etc. we should be specifying those somewhere?

mraspaud commented 1 year ago

yes, you can do that, but it's a bit more complicated, eg:

    from satpy.dataset import DataQuery
    dq = DataQuery(name="S4", stripe="b", view="nadir")
    scn.load([dq])

As you probably know though, not all stripe modes are available for all channels... So S1 has only stripe "a" for example

kameshvinjamuri commented 1 year ago

yeah I know regarding slstr, but regarding satpy my doubt is bit amateurish, how to tell the in the following lines which stripes? any prior configuration to the .py or yml is required after installing through conda?

    files = glob.glob("/home/*/Desktop/mpc/slstr/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/*")
    scn = Scene(filenames=files, reader="slstr_l1b")

mraspaud commented 1 year ago

look at my code in the previous comment, you can specify the stripe in the DataQuery object.

kameshvinjamuri commented 1 year ago

yeah that's in the DataQuery, but the stripe issues pop out in my Scence. dq = DataQuery(name="S4", stripe="b", view="nadir") is fine, since myscene fails I can't load scn.load([dq])

simonrp84 commented 1 year ago

Are these reprocessed files? I can't remember the exact pattern but it looks like it from the filename.

mraspaud commented 1 year ago

what I pasted above works for me with you data @kameshvinjamuri

kameshvinjamuri commented 1 year ago

Are these reprocessed files? I can't remember the exact pattern but it looks like it from the filename.

no, they are Level-1B product geolocated radiometric measurements.

@mraspaud okay thanks, i will recheck everything

kameshvinjamuri commented 1 year ago

@mraspaud everything looks correct what I am doing. just one doubt...in the commands which worked for you (below), you never specified what stripe to take and what viewing direction of the satellite...how does it actually open the bands? is it a random selection? or prior to everything you make the changes to the slstr_l1b reader files?

from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()

if __name__ == '__main__':
    files = find_files_and_readers(base_dir="/home/a001673/data/satellite/Sentinel-3/",
                                   reader='slstr_l1b')
    scn = Scene(filenames=files)

    composite = "overview"
    scn.load([composite])

    nscn = scn.resample('maspalomas')
    nscn.save_dataset(composite)

mraspaud commented 1 year ago

For the color composite, the defaults are used, which is nadir view, and the first available stripe (a for visible channels for example)

kameshvinjamuri commented 1 year ago

hi @mraspaud , it does work when I exclusively create the env for satpy. all is good it seems but geotiff creation, resulting in the foloowong err:

from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()

if __name__ == '__main__':
    files = find_files_and_readers(base_dir="/home/*/slstr/", reader='slstr_l1b')

    scn = Scene(filenames=files, reader='slstr_l1b')
    scn.keys()
    scn.available_dataset_names()
    my_channel = 'S1'
    scn.load([my_channel])
    scn[my_channel].data

    import matplotlib.pyplot as plt
    plt.figure()
    scn[my_channel].plot.imshow(cmap='viridis')
    plt.show()

    #lons, lats = scn[my_channel].attrs['area'].get_lonlats()
    #lats.compute()
    #scn.save_datasets()

    scn.load(["S1"])
    scn.save_dataset('S1', 's1.tif')

[DEBUG: 2023-01-25 14:41:48 : satpy.writers] Reading ['/home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/etc/writers/geotiff.yaml']
/home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2023-01-25 14:41:48 : satpy.writers] Adding enhancement configuration from file: /home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/etc/enhancements/generic.yaml
/home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2023-01-25 14:41:48 : satpy.writers] Data for DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<1>, view=<1>, stripe=<1>, modifiers=()) will be enhanced with options:
        [{'name': 'linear_stretch', 'method': <function stretch at 0x152b7283dbc0>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x152b7283cea0>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2023-01-25 14:41:48 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2023-01-25 14:41:48 : trollimage.xrimage] Applying gamma 1.5
[DEBUG: 2023-01-25 14:41:48 : pyproj] PROJ_DEBUG: proj_is_equivalent_to_with_ctx: /home/*/miniconda3/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation.
[DEBUG: 2023-01-25 14:41:48 : pyproj] PROJ_DEBUG: proj_as_wkt: /home/*/miniconda3/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation.
[INFO: 2023-01-25 14:41:48 : trollimage._xrimage_rasterio] Couldn't create geotransform
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x152b718a5950>
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Starting outermost env
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x152b73b6ee00> created
[DEBUG: 2023-01-25 14:41:48 : rasterio._env] GDAL data files are available at built-in paths.
[DEBUG: 2023-01-25 14:41:48 : rasterio._env] PROJ_LIB found in environment.
[DEBUG: 2023-01-25 14:41:48 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x152b73b6ee00>.
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x152b718a5950>
[DEBUG: 2023-01-25 14:41:48 : rasterio._io] Path: _UnparsedPath(path='s1.tif'), mode: w, driver: GTiff
/home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/rasterio/__init__.py:314: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
  dataset = writer(
[DEBUG: 2023-01-25 14:41:48 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-25 14:41:48 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x152b718a5950>
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x152b73b6ee00> options
[DEBUG: 2023-01-25 14:41:48 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x152b73b6ee00>.
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Exiting outermost env
[DEBUG: 2023-01-25 14:41:48 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x152b718a5950>
/home/*/miniconda3/envs/satpy/lib/python3.11/site-packages/dask/array/chunk.py:276: RuntimeWarning: invalid value encountered in cast
  return x.astype(astype_dtype, **kwargs)
ERROR 1: PROJ: proj_create_from_name: /home/*/miniconda3/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation.

mraspaud commented 1 year ago

Nice that it works at least for reading! The last line of you log seems to indicate a conflict between to PROJ installations?

kameshvinjamuri commented 1 year ago

hi @mraspaud , it came down to this

scn.load(["S1"])
scn.save_dataset('S1', 's1n.tif')

[DEBUG: 2023-01-26 09:39:18 : satpy.writers] Reading ['/home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2023-01-26 09:39:18 : satpy.writers] Adding enhancement configuration from file: /home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2023-01-26 09:39:18 : satpy.writers] Data for DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<calibration.reflectance>, view=<view.nadir>, stripe=<stripe.a>, modifiers=()) will be enhanced with options:
    [{'name': 'linear_stretch', 'method': <function stretch at 0x14b51da31d80>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x14b51da31e10>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2023-01-26 09:39:18 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2023-01-26 09:39:18 : trollimage.xrimage] Applying gamma 1.5
[INFO: 2023-01-26 09:39:18 : trollimage._xrimage_rasterio] Couldn't create geotransform
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x14b53717eda0>
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Starting outermost env
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x14b5371cce20> created
[DEBUG: 2023-01-26 09:39:18 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-01-26 09:39:18 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2023-01-26 09:39:18 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x14b5371cce20>.
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x14b53717eda0>
[DEBUG: 2023-01-26 09:39:18 : rasterio._io] Path: _UnparsedPath(path='s1n.tif'), mode: w, driver: GTiff
[DEBUG: 2023-01-26 09:39:18 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 09:39:18 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x14b53717eda0>
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x14b5371cce20> options
[DEBUG: 2023-01-26 09:39:18 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x14b5371cce20>.
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Exiting outermost env
[DEBUG: 2023-01-26 09:39:18 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x14b53717eda0>

mraspaud commented 1 year ago

ok, does it mean you got it to work?

kameshvinjamuri commented 1 year ago

9@mraspaud not exactly. but I see no error, it forms a tif, no geotiff ( I opened in quantum gis ). i see Couldn't create geotransform in the logs. is this due to bad gdal installation?

mraspaud commented 1 year ago

Good question! What happens if you save to png instead?

kameshvinjamuri commented 1 year ago

unfortunately png is also not georeferenced

pnuu commented 1 year ago

Is SLSTR data a swath, or does it have a proper area definition already in the input files? If not, for geotiff you'll need to resample the data before saving to get georeferenced geotiffs.

mraspaud commented 1 year ago

unfortunately png is also not georeferenced

Indeed, but I was just trying to eliminate possible errors from satpy. So I just wanted to know if it was working for you with pngs. As everything works fine on my laptop with the data I got from you, I'm suspecting something wrong in the environment you are using. Making sure gdal and proj are not installed system-wise but just in the conda environment you are using would be also worth a try I think.

mraspaud commented 1 year ago

Is SLSTR data a swath, or does it have a proper area definition already in the input files? If not, for geotiff you'll need to resample the data before saving to get georeferenced geotiffs.

It is swath, no proper area definition. So resampling is needed for a georeferenced geotiff to be produced.

kameshvinjamuri commented 1 year ago

@mraspaud @pnuu

tried the following, yeah both proj and gdal are independent of system-wise

new_scn = scn.resample(resampler='native')
#new_scn.load(["S1"])
new_scn.save_dataset('S1', '/home/*/s1new.tif')

[DEBUG: 2023-01-26 15:46:06 : satpy.writers] Reading ['/home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2023-01-26 15:46:07 : satpy.writers] Adding enhancement configuration from file: /home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2023-01-26 15:46:07 : satpy.writers] Data for DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<calibration.reflectance>, view=<view.nadir>, stripe=<stripe.a>, modifiers=()) will be enhanced with options:
    [{'name': 'linear_stretch', 'method': <function stretch at 0x147611331e10>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x147611331ea0>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2023-01-26 15:46:07 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2023-01-26 15:46:07 : trollimage.xrimage] Applying gamma 1.5
[INFO: 2023-01-26 15:46:07 : trollimage._xrimage_rasterio] Couldn't create geotransform
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x1476110f30a0>
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Starting outermost env
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x1476110bee00> created
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x1476110bee00>.
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x1476110f30a0>
[DEBUG: 2023-01-26 15:46:07 : rasterio._io] Path: _UnparsedPath(path='S1_20170602_090746.tif'), mode: w, driver: GTiff
[DEBUG: 2023-01-26 15:46:07 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 15:46:07 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x1476110f30a0>
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x1476110bee00> options
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x1476110bee00>.
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exiting outermost env
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x1476110f30a0>
[INFO: 2023-01-26 15:46:07 : satpy.writers] Computing and writing results...
[DEBUG: 2023-01-26 15:46:07 : satpy.scene] Resampling DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<calibration.reflectance>, view=<view.nadir>, stripe=<stripe.a>, modifiers=())
[INFO: 2023-01-26 15:46:07 : satpy.scene] Not reducing data before resampling.
[DEBUG: 2023-01-26 15:46:07 : satpy.writers] Reading ['/home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2023-01-26 15:46:07 : satpy.writers] Adding enhancement configuration from file: /home/*/miniconda3/envs/satpy/lib/python3.10/site-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2023-01-26 15:46:07 : satpy.writers] Data for DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<calibration.reflectance>, view=<view.nadir>, stripe=<stripe.a>, modifiers=()) will be enhanced with options:
    [{'name': 'linear_stretch', 'method': <function stretch at 0x147611331e10>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x147611331ea0>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2023-01-26 15:46:07 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2023-01-26 15:46:07 : trollimage.xrimage] Applying gamma 1.5
[INFO: 2023-01-26 15:46:07 : trollimage._xrimage_rasterio] Couldn't create geotransform
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x147610e79330>
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Starting outermost env
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x147611157e80> created
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x147611157e80>.
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x147610e79330>
[DEBUG: 2023-01-26 15:46:07 : rasterio._io] Path: _UnparsedPath(path='/home/*/s1new.tif'), mode: w, driver: GTiff
[DEBUG: 2023-01-26 15:46:07 : rasterio._io] Skipped delete for overwrite, dataset does not exist: '/home/*/s1new.tif'
[DEBUG: 2023-01-26 15:46:07 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 15:46:07 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x147610e79330>
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x147611157e80> options
[DEBUG: 2023-01-26 15:46:07 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x147611157e80>.
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exiting outermost env
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x147610e79330>

are these lines common for your logs as well? @mraspaud

[DEBUG: 2023-01-26 15:46:07 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-26 15:46:07 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x1476110bee00> created

mraspaud commented 1 year ago

This is what I run:

from satpy import Scene
from satpy.utils import debug_on
import glob
debug_on()

if __name__ == '__main__':
    files = glob.glob("/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/*")
    scn = Scene(filenames=files, reader="slstr_l1b")

    channel = "S1"
    scn.load([channel])
    scn.save_dataset("S1", filename="S1.tif")

At the end I get a S1.tif file in the directory I run the python script in.

Here is the complete log output:

[DEBUG: 2023-01-26 16:07:19 : satpy.readers.yaml_reader] Reading ('/home/a001673/usr/src/satpy/satpy/etc/readers/slstr_l1b.yaml',)
[DEBUG: 2023-01-26 16:07:19 : satpy.readers.yaml_reader] Assigning to slstr_l1b: ['/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S3_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S3_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S2_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_radiance_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_radiance_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_radiance_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S2_radiance_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F1_BT_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S9_BT_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S8_BT_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F1_BT_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F2_BT_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F2_BT_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S8_BT_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S7_BT_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S9_BT_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S7_BT_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geometry_tn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geometry_to.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_tx.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/geodetic_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/flags_bo.nc']
[DEBUG: 2023-01-26 16:07:19 : rasterio.session] Could not import boto3, continuing with reduced functionality.
[WARNING: 2023-01-26 16:07:23 : satpy.readers] Don't know how to open the following files: {'/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_tx.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F1_quality_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F2_quality_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S2_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S8_quality_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/xfdumanifest.xml', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/time_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/time_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S2_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S7_quality_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S3_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/time_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/indices_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/met_tx.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/viscal.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/time_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_bo.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S1_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S9_quality_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F2_quality_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S4_quality_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_cn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_ao.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S5_quality_co.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S3_quality_an.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_bn.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S8_quality_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/cartesian_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/F1_quality_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S9_quality_io.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S7_quality_in.nc', '/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/S6_quality_cn.nc'}
[DEBUG: 2023-01-26 16:07:23 : satpy.composites.config_loader] Looking for composites config file slstr.yaml
[DEBUG: 2023-01-26 16:07:23 : satpy.composites.config_loader] Looking for composites config file visir.yaml
/home/a001673/usr/src/satpy/satpy/_config.py:129: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in cached_entry_points().get(name, []):
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading latitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.yaml_reader] No coordinates found for DataID(name='latitude', resolution=500, view=<view.nadir>, stripe=<stripe.a>, modifiers=())
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading longitude.
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.yaml_reader] No coordinates found for DataID(name='longitude', resolution=500, view=<view.nadir>, stripe=<stripe.a>, modifiers=())
[DEBUG: 2023-01-26 16:07:23 : satpy.readers.slstr_l1b] Reading S1.
[DEBUG: 2023-01-26 16:07:23 : satpy.writers] Reading ['/home/a001673/usr/src/satpy/satpy/etc/writers/geotiff.yaml']
/home/a001673/usr/src/satpy/satpy/_config.py:129: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in cached_entry_points().get(name, []):
[DEBUG: 2023-01-26 16:07:24 : satpy.writers] Adding enhancement configuration from file: /home/a001673/usr/src/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2023-01-26 16:07:24 : satpy.writers] Adding enhancement configuration from file: /home/a001673/usr/src/spy-test/enhancements/generic.yaml
[DEBUG: 2023-01-26 16:07:24 : satpy.writers] Data for DataID(name='S1', wavelength=WavelengthRange(min=0.545, central=0.555, max=0.565, unit='µm'), resolution=500, calibration=<calibration.reflectance>, view=<view.nadir>, stripe=<stripe.a>, modifiers=()) will be enhanced with options:
    [{'name': 'linear_stretch', 'method': <function stretch at 0x7f2dc5968940>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x7f2dc5968a60>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2023-01-26 16:07:24 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2023-01-26 16:07:24 : trollimage.xrimage] Applying gamma 1.5
[INFO: 2023-01-26 16:07:24 : trollimage._xrimage_rasterio] Couldn't create geotransform
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f2dc6bad1b0>
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Starting outermost env
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f2dc6bacdc0> created
[DEBUG: 2023-01-26 16:07:24 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2023-01-26 16:07:24 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-01-26 16:07:24 : rasterio._env] PROJ_LIB found in environment.
[DEBUG: 2023-01-26 16:07:24 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f2dc6bacdc0>.
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f2dc6bad1b0>
[DEBUG: 2023-01-26 16:07:24 : rasterio._io] Path: _UnparsedPath(path='S1.tif'), mode: w, driver: GTiff
/home/a001673/mambaforge/envs/satpy/lib/python3.10/site-packages/rasterio/__init__.py:287: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
  dataset = writer(
[DEBUG: 2023-01-26 16:07:24 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 16:07:24 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f2dc6bad1b0>
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f2dc6bacdc0> options
[DEBUG: 2023-01-26 16:07:24 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f2dc6bacdc0>.
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Exiting outermost env
[DEBUG: 2023-01-26 16:07:24 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f2dc6bad1b0>
/home/a001673/mambaforge/envs/satpy/lib/python3.10/site-packages/dask/array/chunk.py:276: RuntimeWarning: invalid value encountered in cast
  return x.astype(astype_dtype, **kwargs)

mraspaud commented 1 year ago

By the way, resampler="native" will not help getting the tiff file georeferenced, since this is swath data. You could try instead to resample to dynamic area that will match your data and be georeferenced, eg:

from satpy.utils import debug_on
import glob
debug_on()

if __name__ == '__main__':
    files = glob.glob("/home/a001673/data/satellite/Sentinel-3/current/S3A_SL_1_RBT____20170602T090746_20170602T091046_20181003T205946_0179_018_207______LR1_R_NT_003.SEN3/*")
    scn = Scene(filenames=files, reader="slstr_l1b")

    channel = "S1"
    scn.load([channel])

    nscn = scn.resample('omerc_bb')
    nscn.save_dataset(channel, filename=f"{channel}.tif")

kameshvinjamuri commented 1 year ago

hi, sorry, it's been a while working on this, everything worked fine, thanks for this except for the stripe for F1

from satpy.dataset import DataQuery
dq = DataQuery(name="F1", stripe="i", view="nadir")
scn.load([dq])
KeyError: "Unknown datasets: {DataQuery(name='F1', stripe='i', view='nadir')}"

ameraner commented 3 months ago

Hi @kameshvinjamuri The F1 channel can have only the stripe f as per design/configuration (see e.g. https://sentiwiki.copernicus.eu/web/slstr-products)

kameshvinjamuri commented 3 months ago

Hi @kameshvinjamuri The F1 channel can have only the stripe f as per design/configuration (see e.g. https://sentiwiki.copernicus.eu/web/slstr-products)

yeah makes sense! (y)

pytroll / satpy

satpy slstr l1b reading issue #2365

Readers

Writers

Extras