search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.04k stars 287 forks source link

geotiff writer fails with dask distributed #2834

Closed gerritholl closed 1 week ago

gerritholl commented 1 week ago

Describe the bug

The geotiff writer fails when using the dask distributed scheduler.

To Reproduce

from dask.distributed import Client, LocalCluster
from glob import glob
from satpy import Scene
from satpy.utils import debug_on; debug_on()

seviri_files = glob("/media/nas/x21308/scratch/SEVIRI/202103300900/H-000*")

def main():
    cluster = LocalCluster()
    client = Client(cluster)
    sc = Scene(filenames=seviri_files, reader=["seviri_l1b_hrit"])
    sc.load(["IR_108"])
    sc.save_datasets("geotiff")

if __name__ == "__main__":
    main()

Expected behavior

I expect an image to be produced without error messages.

Actual results

Computing writer results fails due to a rasterio exception: TypeError: self._hds cannot be converted to a Python object for pickling

[DEBUG: 2024-06-20 15:42:47 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : asyncio] Using selector: EpollSelector
[DEBUG: 2024-06-20 15:42:48 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/seviri_l1b_hrit.yaml',)
[DEBUG: 2024-06-20 15:42:48 : satpy.readers.yaml_reader] Assigning to seviri_l1b_hrit: ['/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000024___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000014___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000012___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000022___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000011___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000016___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000015___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000009___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000021___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000010___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000023___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000017___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000020___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000019___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000013___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000018___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-HRV______-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_016___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_039___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_087___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_097___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_108___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_120___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-IR_134___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS006___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-VIS008___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_062___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000003___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000002___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000006___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000007___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000004___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000001___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000005___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-WV_073___-000008___-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-_________-PRO______-202103300900-__', '/media/nas/x21308/scratch/SEVIRI/202103300900/H-000-MSG4__-MSG4________-_________-EPI______-202103300900-__']
[INFO: 2024-06-20 15:42:48 : hrit_msg] No IMPF configuration field found in prologue.
[DEBUG: 2024-06-20 15:42:48 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2024-06-20 15:42:48 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/pyorbital/etc
[DEBUG: 2024-06-20 15:42:48 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.readers.yaml_reader] Requested orientation for Dataset None is 'native' (default). No flipping is applied.
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /home/gholl/Arbeit/checkouts-perforce/config/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2024-06-20 15:42:49 : satpy.writers] Data for DataID(name='IR_108', wavelength=WavelengthRange(min=9.8, central=10.8, max=11.8, unit='µm'), resolution=3000.403165817, calibration=<2>, modifiers=()) will be enhanced with options:
    [{'name': 'stretch', 'method': <function stretch at 0x7f73bb20f600>, 'kwargs': {'stretch': 'linear'}}]
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2024-06-20 15:42:49 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f73bb4aaef0> created
[DEBUG: 2024-06-20 15:42:49 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f73bb4aaef0>.
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio._io] Path: _UnparsedPath(path='IR_108_20210330_090000.tif'), mode: w, driver: GTiff
[DEBUG: 2024-06-20 15:42:49 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-20 15:42:49 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f740c109a00>
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f73bb4aaef0> options
[DEBUG: 2024-06-20 15:42:49 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f73bb4aaef0>.
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-20 15:42:49 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f740c109a00>
[INFO: 2024-06-20 15:42:49 : satpy.writers] Computing and writing results...
2024-06-20 15:42:49,450 - distributed.protocol.pickle - ERROR - Failed to serialize <ToPickle: HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7f73bb25f560>
 0. store-sources-03ce331851719a2514a22c56dc4ea25e
 1. store-map-9809be199e75ba2d686f687a2db7a555
 2. store-4b68c2333fe6b23c4552346ea1966e10
>.
Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling
Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 363, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 78, in pickle_dumps
    frames[0] = pickle.dumps(
                ^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "<stringsource>", line 2, in rasterio._io.DatasetWriterBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/gholl/checkouts/protocode/test-dask-distributed-geotiff.py", line 16, in <module>
    main()
  File "/data/gholl/checkouts/protocode/test-dask-distributed-geotiff.py", line 13, in main
    sc.save_datasets("geotiff")
  File "/data/gholl/checkouts/satpy/satpy/scene.py", line 1293, in save_datasets
    return writer.save_datasets(dataarrays, compute=compute, **save_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/writers/__init__.py", line 756, in save_datasets
    return compute_writer_results([results])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/writers/__init__.py", line 594, in compute_writer_results
    da.compute(delayeds)
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/dask/base.py", line 661, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/distributed/protocol/serialize.py", line 389, in serialize
    raise TypeError(msg, str_x) from exc
TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 3 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x7f73bb25f560>\n 0. store-sources-03ce331851719a2514a22c56dc4ea25e\n 1. store-map-9809be199e75ba2d686f687a2db7a555\n 2. store-4b68c2333fe6b23c4552346ea1966e10\n>')

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

Additional context

Encountered while working on #2686.

djhoese commented 1 week ago

This is a known limitation of the writer. We need to switch to rioxarray and use more complicated locking to allow for threaded or distributed schedulers.

gerritholl commented 1 week ago

Duplicate of #1762.