TypeError: cannot pickle '_thread.lock' object

pradeeps31 commented 1 year ago

Discussed in https://github.com/corteva/rioxarray/discussions/613

^{Originally posted by **pradeeps31** December 8, 2022} Hi Guys, I am trying to convert a netCDF file to GeoTIFF file using xarrray and rioxarray. But when I give the rio.to_raster command, it throws the TypeError: cannot pickle '_thread.lock' object. The NetCDF file (.nc4) is a precipitation dataset downloaded from NASA Earthdata. The code I have used is below: ``` >>> import xarray as xr >>> import rioxarray as rio >>> ncfile = xr.open_dataset('3B-DAY.MS.MRG.3IMERG.20210901-S000000-E235959.V06.nc4') >>> pr = ncfile['precipitationCal'] >>> pr = pr.transpose('time', 'lat', 'lon') >>> pr = pr.rio.set_spatial_dims('lon', 'lat') >>> pr.rio.crs >>> pr.rio.crs >>> pr.rio.set_crs("epsg:4326") [6480000 values with dtype=float32] Coordinates: * lon (lon) float32 -179.9 -179.8 -179.8 -179.6 ... 179.8 179.9 179.9 * lat (lat) float32 -89.95 -89.85 -89.75 -89.65 ... 89.75 89.85 89.95 * time (time) object 2021-09-01 00:00:00 Attributes: units: mm long_name: Daily accumulated precipitation (combined microwave-IR) estimate >>> pr.rio.to_raster(r"trial1.tif") Traceback (most recent call last): File "", line 1, in File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_array.py", line 1145, in to_raster return RasterioWriter(raster_path=raster_path).to_raster( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_writer.py", line 232, in to_raster xarray_dataarray = xarray_dataarray.copy() ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1181, in copy return self._copy(deep=deep, data=data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1189, in _copy variable = self.variable._copy(deep=deep, data=data, memo=memo) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\variable.py", line 1048, in _copy ndata = copy.deepcopy(ndata, memo) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy y = copier(x, memo) ^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] ^^^^^^^^^^^^^^^^^ File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 161, in deepcopy rv = reductor(4) ^^^^^^^^^^^ TypeError: cannot pickle '_thread.lock' object >>> ``` Could anyone help me to resolve this issue. I'm using python 3.11 version. Thanks Pradeep

snowman2 commented 1 year ago

Mind filling out the issue template:

Environment Information

python -c "import rioxarray; rioxarray.show_versions()"

rioxarray version (python -c "import rioxarray; print(rioxarray.__version__)")
rasterio version (rio --version)
GDAL version (rio --gdal-version)
Python version (python -c "import sys; print(sys.version.replace('\n', ' '))")
Operation System Information (python -c "import platform; print(platform.platform())")

Installation method

conda, pypi, from source, etc...

Conda environment information (if you installed with conda):

Environment (conda list):

``` $ conda list | grep -E "rasterio|xarray|gdal" ```

Details about conda and system ( conda info ):

``` $ conda info ```

pradeeps31 commented 1 year ago

Environment Information:

rioxarray (0.13.1) deps:
  rasterio: 1.3.4
    xarray: 2022.12.0
      GDAL: 3.6.0
      GEOS: 3.11.1
      PROJ: 9.1.0
 PROJ DATA: C:\Users\Pradeep\anaconda3\envs\imerg_env\Library\share\proj
 GDAL DATA: C:\Users\Pradeep\anaconda3\envs\imerg_env\Library\share\gdal

Other python deps:
     scipy: 1.9.3
    pyproj: 3.4.0

System:
    python: 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:12:32) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\Pradeep\anaconda3\envs\imerg_env\python.exe
   machine: Windows-10-10.0.19045-SP0

Installation method: conda

Environment (conda list):

libgdal                   3.6.0               hc44c3d3_11    conda-forge
rasterio                  1.3.4           py311hd28ea1e_0    conda-forge
rioxarray                 0.13.1             pyhd8ed1ab_0    conda-forge
xarray                    2022.12.0          pyhd8ed1ab_0    conda-forge

Details about conda and system ( conda info ):

   active environment : imerg_env
    active env location : C:\Users\Pradeep\anaconda3\envs\imerg_env
            shell level : 2
       user config file : C:\Users\Pradeep\.condarc
 populated config files : C:\Users\Pradeep\.condarc
          conda version : 22.9.0
    conda-build version : 3.21.8
         python version : 3.9.12.final.0
       virtual packages : __cuda=9.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\Pradeep\anaconda3  (writable)
      conda av data dir : C:\Users\Pradeep\anaconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\Pradeep\anaconda3\pkgs
                          C:\Users\Pradeep\.conda\pkgs
                          C:\Users\Pradeep\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\Pradeep\anaconda3\envs
                          C:\Users\Pradeep\.conda\envs
                          C:\Users\Pradeep\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/22.9.0 requests/2.27.1 CPython/3.9.12 Windows/10 Windows/10.0.19045
          administrator : False
             netrc file : None
           offline mode : False

snowman2 commented 1 year ago

This one may be more difficult to track down. I don't have a Windows machine and the CICD runners don't show this issue. If someone else runs into this, more details are welcome.

pradeeps31 commented 1 year ago

Hi @snowman2,

I was also unable to find exactly the cause of that error. But, I was able to find a workaround for this issue. I created an environment with the python 3.7.12 version and was able to run the code. But I would like to know what was the source of the error and the solution to the same.

swesemeyer commented 1 year ago

I have the same issue on Fedora Linux 38. test_prec.nc.zip

Here is the code:

    xds = xarray.open_dataset("/tmp/test_prec.nc")
    precip = xds.precipitationCal.transpose('time', 'lat', 'lon')
    precip.rio.set_spatial_dims(x_dim="lon", y_dim="lat", inplace=True)
    precip.rio.write_crs("EPSG:4326", inplace=True)
    precip.rio.to_raster("/tmp/test_prec.tif") <-- this fails with the same error message

Error: File "/usr/lib64/python3.10/copy.py", line 161, in deepcopy rv = reductor(4) TypeError: cannot pickle '_thread.lock' object

rioxarray.show_versions() rioxarray (0.14.1) deps: rasterio: 1.3.6 xarray: 2023.4.2 GDAL: 3.5.3 GEOS: 3.11.1 PROJ: 9.0.1 PROJ DATA: /home/user/venv_310/lib64/python3.10/site-packages/rasterio/proj_data GDAL DATA: /home/user/venv_310/lib64/python3.10/site-packages/rasterio/gdal_data Other python deps: scipy: 1.10.1 pyproj: 3.4.1 System: python: 3.10.11 (main, Apr 5 2023, 00:00:00) [GCC 13.0.1 20230404 (Red Hat 13.0.1-0)] executable: /home/user/venv_310/bin/python machine: Linux-6.2.15-300.fc38.x86_64-x86_64-with-glibc2.37

snowman2 commented 2 weeks ago

https://github.com/pydata/xarray/pull/8571 was included in v2024.01.0 and says it addresses the issue.

corteva / rioxarray