search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.04k stars 287 forks source link

Sentinel-1 sar-c safe reader consumes too much memory and a lot of time #2830

Closed yukaribbba closed 1 week ago

yukaribbba commented 1 week ago

Describe the bug Nearly all my 64GB memory and it still asks for more.

To Reproduce

files = find_files_and_readers(base_dir="C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558",
                               reader="sar-c_safe",
                               )
scn = Scene(filenames=files)
scn.load(["green-sar"])
scn.save_dataset("green-sar", filename="green.tif", fill_value=0, compress=None)

Actual results

C:\Users\45107\miniforge3\python.exe C:\Users\45107\PycharmProjects\Satellite\testcode.py 
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\readers\\sar-c_safe.yaml',)
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\readers\\sar-c_safe.yaml',)
[WARNING: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Available datasets are unknown, returning all datasets...
[DEBUG: 2024-06-18 23:25:29 : satpy.composites.config_loader] Looking for composites config file sar-c.yaml
[DEBUG: 2024-06-18 23:25:29 : satpy.composites.config_loader] Looking for composites config file sar.yaml
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015936B1E410> created
[DEBUG: 2024-06-18 23:25:30 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015936B1E410>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Sharing flag: 0
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558/S1A_IW_GRDH_1SDV_20240614T095536_20240614T095558_054318_069B94_1707.SAFE/measurement/s1a-iw-grd-vv-20240614t095536-20240614t095558-054318-069b94-001.tiff' mode='r'> is started.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015936B1E410> options
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015936B1E410>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015936FAE020> created
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015936FAE020>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Sharing flag: 0
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558/S1A_IW_GRDH_1SDV_20240614T095536_20240614T095558_054318_069B94_1707.SAFE/measurement/s1a-iw-grd-vh-20240614t095536-20240614t095558-054318-069b94-002.tiff' mode='r'> is started.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015936FAE020> options
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015936FAE020>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vv', resolution=80, calibration=<1>, quantity=<1>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vh', resolution=80, calibration=<1>, quantity=<1>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vv', resolution=80, calibration=<1>, quantity=<2>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Reading ['C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\writers\\geotiff.yaml']
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Adding enhancement configuration from file: C:\Users\45107\Documents\GitHub\satpy\satpy\etc\enhancements\generic.yaml
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Adding enhancement configuration from file: D:\satpy_config\enhancements\generic.yaml
[DEBUG: 2024-06-18 23:25:31 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Data for DataID(name='green-sar', resolution=80) will be enhanced with options:
    [{'name': 'stretch', 'method': <function stretch at 0x0000015936F3A700>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0.00109, -17.57, 0.00332], 'max_stretch': [0.0594, -3.3, 0.3]}}]
[DEBUG: 2024-06-18 23:25:31 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0.00109, -17.57, 0.00332], 'max_stretch': [0.0594, -3.3, 0.3]}
[DEBUG: 2024-06-18 23:25:31 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015937073460> created
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015937073460>.
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Path: _UnparsedPath(path='green.tif'), mode: w, driver: GTiff
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Skipped delete for overwrite, dataset does not exist: 'green.tif'
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015937073460> options
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015937073460>.
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=14586, width=25294, height=473)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=14586.0 width=25294.0 height=473.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=14586, width=25294, height=473)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=14586.0 width=25294.0 height=473.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=13260, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=13260.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=13260, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=13260.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=0, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=0.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=0, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=0.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=1326, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=1326.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=1326, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=1326.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=2652, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=2652.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=2652, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=2652.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=3978, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=3978.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=3978, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=3978.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Window: Window(col_off=0, row_off=5304, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] IO window xoff=0.0 yoff=5304.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Window: Window(col_off=0, row_off=5304, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] IO window xoff=0.0 yoff=5304.0 width=25294.0 height=1326.0
Traceback (most recent call last):
  File "C:\Users\45107\PycharmProjects\Satellite\testcode.py", line 24, in <module>
    scn.save_dataset("green-sar", filename="green.tif", fill_value=0, compress=None)
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\scene.py", line 1231, in save_dataset
    return writer.save_dataset(self[dataset_id],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\writers\__init__.py", line 885, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\writers\geotiff.py", line 278, in save_image
    return img.save(filename, fformat="tif", driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\trollimage\xrimage.py", line 248, in save
    return self.rio_save(filename, fformat=fformat, driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\trollimage\xrimage.py", line 432, in rio_save
    res = da.store(*to_store)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\array\core.py", line 1229, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\base.py", line 402, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\threaded.py", line 90, in get
    results = get_async(
              ^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 512, in get_async
    raise_exception(exc, tb)
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 320, in reraise
    raise exc
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 225, in execute_task
    result = _execute_task(task, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\optimization.py", line 1001, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 157, in get
    result = _execute_task(task, cache)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\utils.py", line 78, in apply
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\readers\sar_c_safe.py", line 508, in intp
    return interpolator((grid_y, grid_x))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "interpnd.pyx", line 192, in scipy.interpolate.interpnd.NDInterpolatorBase.__call__
  File "interpnd.pyx", line 333, in scipy.interpolate.interpnd.LinearNDInterpolator._evaluate_double
  File "interpnd.pyx", line 349, in scipy.interpolate.interpnd.LinearNDInterpolator._do_evaluate
  File "interpnd.pyx", line 172, in scipy.interpolate.interpnd.NDInterpolatorBase._find_simplicies
  File "C:\Users\45107\miniforge3\Lib\site-packages\numpy\lib\function_base.py", line 962, in copy
    return array(a, order=order, subok=subok, copy=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 16.0 GiB for an array with shape (33539844, 64) and data type float64

Environment Info:

mraspaud commented 1 week ago

Thanks for reporting this issue. Indeed, the sentinel1 data is very heavy, and I have to take small chunk sizes when working with that data on my laptop, for example:

DASK_ARRAY__CHUNK_SIZE=32MB my_s1_script.py

See more on how to tweak performance here: https://satpy.readthedocs.io/en/stable/faq.html#why-is-satpy-slow-on-my-powerful-machine

yukaribbba commented 1 week ago

Thanks for the hint and it does work for me. I usually set DASK_ARRAY__CHUNK_SIZE to 2200*2200 (about 68MB.), which gives me satisfying performance on most other readers except this one. I wish we had more specific tips for adjust the value depending on readers. That'd be better...

mraspaud commented 1 week ago

I agree, we should give more information when we can. Feel free to create a PR about it if you want.

yukaribbba commented 1 week ago

Ok that's on my todo list. For now I'll just close this.