2020/08/01/NetCDF-to-GeoTIFF-using-Python

[utterances-bot]

NetCDF to GeoTIFF using Python - Pratiman Homepage

This is Pratiman webpage with some blogs.

https://pratiman-91.github.io/2020/08/01/NetCDF-to-GeoTIFF-using-Python.html

[e5k]

Nice and easy - thank you

[dncardenasg]

Thanks. How could I do the same for multiple netcdf files in one directory? I have tried, but have not succeeded.

[pratiman-91]

If you have same multiple netcdf files in a directory then you can loop through each file. For example:

from glob import glob
import xarray as xr
import rioxarray as rio

path = '/<directory path>/' # Change here
for file in glob(path + '*.nc'):
    print("Processing: " + file)
    ncfile = xr.open_dataset(file)

    #Extract the variable
    pr = ncfile['pr']

    #(Optional) convert longitude from (0-360) to (-180 to 180) (if required)
    pr.coords['lon'] = (pr.coords['lon'] + 180) % 360 - 180
    pr = pr.sortby(pr.lon)

    #Define lat/long 
    pr = pr.rio.set_spatial_dims('lon', 'lat')

    #Check for the CRS
    pr.rio.crs

    #(Optional) If your CRS is not discovered, you should be able to add it like so:
    pr.rio.set_crs("epsg:4326")

    #Saving the file
    pr.rio.to_raster(file[:-2] + '.tif')

[dncardenasg]

Thanks a lot!!

[praveenkalura]

Hi! I want to convert Multiple tiff files in folder to one netcdf. Can you provide code for it?

[pradeeps31]

Hi, I am trying to convert from NetCDF (.nc4) file to GeoTIFF for IMERG data downloaded from NASA earth data. I am following the same procedure in your code but it throws the below error:

>>> import xarray as xr
>>> import rioxarray as rio
>>> ncfile=xr.open_dataset('3B-DAY.MS.MRG.3IMERG.20210901-S000000-E235959.V06.nc4')
>>> ncfile
<xarray.Dataset>
Dimensions:                    (time: 1, lon: 3600, lat: 1800, nv: 2)
Coordinates:
  * lon                        (lon) float32 -179.9 -179.8 ... 179.9 179.9
  * lat                        (lat) float32 -89.95 -89.85 ... 89.85 89.95
  * time                       (time) object 2021-09-01 00:00:00
Dimensions without coordinates: nv
Data variables:
    precipitationCal           (time, lon, lat) float32 ...
    precipitationCal_cnt       (time, lon, lat) int8 ...
    precipitationCal_cnt_cond  (time, lon, lat) int8 ...
    HQprecipitation            (time, lon, lat) float32 ...
    HQprecipitation_cnt        (time, lon, lat) int8 ...
    HQprecipitation_cnt_cond   (time, lon, lat) int8 ...
    randomError                (time, lon, lat) float32 ...
    randomError_cnt            (time, lon, lat) int8 ...
    time_bnds                  (time, nv) object ...
Attributes:
    BeginDate:       2021-09-01
    BeginTime:       00:00:00.000Z
    EndDate:         2021-09-01
    EndTime:         23:59:59.999Z
    FileHeader:      StartGranuleDateTime=2021-09-01T00:00:00.000Z;\nStopGran...
    InputPointer:    3B-HHR.MS.MRG.3IMERG.20210901-S000000-E002959.0000.V06B....
    title:           GPM IMERG Final Precipitation L3 1 day 0.1 degree x 0.1 ...
    DOI:             10.5067/GPM/IMERGDF/DAY/06
    ProductionTime:  2021-12-28T12:50:10.716Z
>>> pr=ncfile['precipitationCal']
>>> pr=pr.rio.set_spatial_dims('lon', 'lat')
>>> pr.rio.crs
>>> pr.rio.set_crs("epsg:4326")
<xarray.DataArray 'precipitationCal' (time: 1, lon: 3600, lat: 1800)>
[6480000 values with dtype=float32]
Coordinates:
  * lon      (lon) float32 -179.9 -179.8 -179.8 -179.6 ... 179.8 179.9 179.9
  * lat      (lat) float32 -89.95 -89.85 -89.75 -89.65 ... 89.75 89.85 89.95
  * time     (time) object 2021-09-01 00:00:00
Attributes:
    units:      mm
    long_name:  Daily accumulated precipitation (combined microwave-IR) estimate
>>> pr.rio.to_raster(r"trial.tif")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Pradeep\anaconda3\envs\imerg\lib\site-packages\rioxarray\raster_array.py", line 1151, in to_raster
    count=int(self.count),
  File "C:\Users\Pradeep\anaconda3\envs\imerg\lib\site-packages\rioxarray\rioxarray.py", line 947, in count
    extra_dim = self._check_dimensions()
  File "C:\Users\Pradeep\anaconda3\envs\imerg\lib\site-packages\rioxarray\rioxarray.py", line 926, in _check_dimensions
    raise InvalidDimensionOrder(
rioxarray.exceptions.InvalidDimensionOrder: Invalid dimension order. Expected order: ('time', 'lat', 'lon'). You can use `DataArray.transpose('time', 'lat', 'lon')` to reorder your dimensions. Data variable: precipitationCal
>>>

I am trying to make a GeoTIFF file for the variable precipitationCal for the measured precipitation. I have tried the transpose function, but even after that, the same error persists.

Kindly request your help with this

Pradeep

[pratiman-91]

@pradeeps31 You can try using pr = pr.transpose('time', 'lat', 'lon') after pr=ncfile['precipitationCal']

This should work!

import xarray as xr
import rioxarray as rio

ncfile = xr.open_dataset('3B-DAY.MS.MRG.3IMERG.20210901-S000000-E235959.V06.nc4')

pr = ncfile['precipitationCal']

pr = pr.transpose('time', 'lat', 'lon')

pr = pr.rio.set_spatial_dims('lon', 'lat')

pr.rio.crs

pr.rio.set_crs("epsg:4326")

pr.rio.to_raster(r"trial.tif")

[GIS243]

Thanks @pratiman-91 c

Is it possible to convert the netcdf file into multiple tiff file(s) , tiff file for each time in NetCDF.

Thanks

[pratiman-91]

@pradeeps31

Is it how to convert the netcdf file into multiple tiff file(s) , tiff file for each time in NetCDF.

No, for each time step you have to loop it through time dimension to generate the GeoTIFF. This code will generate different bands for the timesteps. Meaning you will get a multi-band raster where each band will be your timestep.

[pradeeps31]

Thank you @pratiman-91 for your quick response. It solved the error for invalid dimensions error. But a different error comes up after giving the _rio.toraster option. Could you kindly have a look a this error and suggest me a solution

>>> pr.rio.to_raster(r"trial1.tif")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_array.py", line 1145, in to_raster
    return RasterioWriter(raster_path=raster_path).to_raster(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\rioxarray\raster_writer.py", line 232, in to_raster
    xarray_dataarray = xarray_dataarray.copy()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1181, in copy
    return self._copy(deep=deep, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\dataarray.py", line 1189, in _copy
    variable = self.variable._copy(deep=deep, data=data, memo=memo)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\site-packages\xarray\core\variable.py", line 1048, in _copy
    ndata = copy.deepcopy(ndata, memo)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\Pradeep\anaconda3\envs\imerg_env\Lib\copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.lock' object

Thanks

Pradeep

[GIS243]

@pradeeps31

Is it how to convert the netcdf file into multiple tiff file(s) , tiff file for each time in NetCDF.

No, for each time step you have to loop it through time dimension to generate the GeoTIFF. This code will generate different bands for the timesteps. Meaning you will get a multi-band raster where each band will be your timestep.

Thanks can you please help with that, how to create a loop

Thanks

[pratiman-91]

@pradeeps31

I am unable to replicate your error. I have tested this code with rioxarray version 0.12.3. I would suggest to use that version.

@GIS243 for looping the time dimension: it may look like something like this

import xarray as xr
import rioxarray as rio

ncfile = xr.open_dataset('3B-DAY.MS.MRG.3IMERG.20210901-S000000-E235959.V06.nc4')

pr = ncfile['precipitationCal']

pr = pr.transpose('time', 'lat', 'lon')
for i in range(len(pr.time)):

    pr1 = pr[i].rio.set_spatial_dims('lon', 'lat')

    pr1.rio.crs

    pr1.rio.set_crs("epsg:4326")

    pr1.rio.to_raster(r"trial_" + str(i) + ".tif")

[GIS243]

@pratiman-91

Thanks

I have tried the code but getting the following error message and there is not tiff file generated please

Traceback (most recent call last): File "h:\netcdf\PM2.5\NETCDF_to_GeoTIFF.py", line 17, in pr1.rio.toraster(r"trial" + str(i) + ".tif") File "C:\Users\ramzan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rioxarray\raster_array.py", line 1145, in to_raster return RasterioWriter(raster_path=raster_path).to_raster( File "C:\Users\ramzan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rioxarray\raster_writer.py", line 232, in to_raster xarray_dataarray = xarray_dataarray.copy() File "C:\Users\ramzan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\xarray\core\dataarray.py", line 1180, in copy return self._copy(deep=deep, data=data) File "C:\Users\ramzan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\xarray\core\dataarray.py", line 1188, in _copy variable = self.variable._copy(deep=deep, data=data, memo=memo) File "C:\Users\ramzan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\xarray\core\variable.py", line 1048, in _copy ndata = copy.deepcopy(ndata, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\copy.py", line 271, in _reconstruct state = deepcopy(state, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 231, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in _deepcopy_tuple y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 211, in y = [deepcopy(a, memo) for a in x] File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64qbz5n2kfra8p0\lib\copy.py", line 161, in deepcopy rv = reductor(4) TypeError: cannot pickle '_thread.lock' object

[pradeeps31]

Hi @pratiman-91, I am also unable to find exactly the cause of that error. But, I was able to find a workaround for this issue. I created an environment with the python 3.7.12 version and was able to run the code. Thank you for your responses. @GIS243 You could also try the same with the 3.7.12 version and see if it works for you.

[GIS243]

@pratiman-91 Thanks

I have tried on linux and it had only generated one file named trial_0.tif

[take-gogogo]

hi, I am going to convert AMSR2 L1B .nc data to geotiff, but ihave a trouble. It does not work well because the coordinate information is in two dimensions. What should I do in this case?

I got AMSR2 data here(https://gportal.jaxa.jp/gpr/search?tab=1)

AMSR2 data stracture is shown below ///////////////////////////////////////////////////////////////////////////////////////////// xarray.Dataset Dimensions: phony_dim_0: 66phony_dim_1: 243phony_dim_2: 486phony_dim_3: 65 Coordinates: lon (phony_dim_0, phony_dim_1) float32 ... lat (phony_dim_0, phony_dim_1) float32 ... Data variables: (21) Indexes: (0) Attributes: selected area : POLYGON((179.765 69.907,-161.075 69.907,-161.075 64.405,179.765 64.405,179.765 69.907)) //////////////////////////////////////////////////////////////////////////////////////////////

I have tried to decrise from 2D to 1D, but it did not work. Any advice you can give me would be great.

[pratiman-91]

@take-gogogo

Convert 2d lat/lon into a 1D, and then process the files using pr[i].rio.set_spatial_dims('lon', 'lat') is not recommended as you will loose information and/or may get wrong information.

It is my understanding that GeoTIFF cannot handle variable grids sizes that is generally represented by 2D lat/lon values. It is best if you use native NetCDF format.

[take-gogogo]

@pratiman-91

thanks for replying. I see.

I would like to reproject this data from EPSG:4326 to EASE-grid 2.0 with bilinear. So I tried to convert this data to Geotiff with rioxarray and xarray, and use gdal to reproject it with bilinear. But after your opinion, I think it is difficult. would you tell me how to reproject the data, which has 2D dimentional lat/lon, with bilinear?

This may be a throwaway question. Best regards.

[pratiman-91]

@take-gogogo

i generally use cdo.

for more information, you can refer to https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-6660002.12.1

[take-gogogo]

@pratiman-91

I tried cdo with different data than AMSR2. This data is given in a separate NETCDF with observations and geographic information. I have tried to add the geographic information according to this site(https://code.mpimet.mpg.de/boards/1/topics/9593) and the instructions for downloading cdo(https://www.youtube.com/watch?v=7pwhEedzh34), but have been unsuccessful.

""" cdo remapbil,latlonmap.nc ArcLeads_20150101.nc out.nc cdo remapbil (Abort): NetCDF: Unknown file format """

We believe that NETCDF may not be loaded. What should we do?

You can download the data here https://drive.google.com/drive/folders/14Wf3TOm4p1bH5c0tJmxMMm0pG19CR7ei?usp=sharing

I originally downloaded the data from here https://doi.pangaea.de/10.1594/PANGAEA.854411

best regards

[Oemameloe]

Hi @pratiman-91

I used your suggested code for generating time dimensional tiff files which you explained to GIS243 a month ago. I used 'tasmax_day_HadGEM3-GC31-MM_ssp585_r1i1p1f3_gn_20350101-20391230.nc' from cmip6. This file can be downloaded from https://handle-esgf.dkrz.de/lp/21.14100/4c6dd795-5d2e-3d3c-b2bf-2bc69bfa7fa9

But when I wanted to do zonal statistics for each of the tiff files with some shapefiles, but it did not work in python. So I checked to see how the tiff files looked like and in Qgis there was nothin wrong and even the zonal statistics worked. But I want to do it in python with which I can loop through all tiff files. So also plotted the tiff file in python and saw it was flipped. The plot was flipped, but the shapefiles which I plotted onto the tiff file where flipped as well, which means the result of the zonal statistics would have been the same if they were not flipped. But then my questions are why does it not work in python and why does it work in GIS? How can I fix the tiff files in a way that the zonal statistics do work in python(for example flipping the tiff files??)? Or is there another way to get zonal statistics without generating tiff files for example with the original NETCDF(for me zonal statistics only worked with tiff files and not yet with NETCDF)? If it works for NETCDF how can I do zonal statistics for every timestep? Best regards

[pratiman-91]

@take-gogogo

Maybe you should ask cdo user forum. It is my guess that something is wrong with the installation.

@Oemameloe Zonal stats in python is still infancy. There are two python packages that might be helpful to you are rasterstats (https://pythonhosted.org/rasterstats/) and wradlib (https://docs.wradlib.org/en/stable/zonalstats.html). In addition, if you want to batch process, then you can also create your script on QGIS. If you really want to process everything in python then you can use xarray to create a custom script for zonal stats.

[pratiman-91]

@anhntv4nk

ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy', 'rasterio']

You need to install backends to open or save the NetCDF file.

For netcdf4 you can install it using pip install netCDF4 or you can also install scipy using pip install scipy.

[scottishaccent]

@pratiman-91 Thanks for doing this! When I ran this script with a floating-point variable from the netcdf file, the resulting geotiff files were created with signed integer values. Do you know how to prevent this from happening?

[scottishaccent]

@pratiman-91 Never mind, my problem was that I forgot to apply the scale factor and offset from the netcdf file.

[DevUt]

Hey thanks alot lot this @pratiman-91. I had netcdf which had multiple bands and each band has time data. I'm only interested in one time for all the bands. Is there a way to combine the 3 bands into a single geotiff

[JuliusBamah]

@pratiman-91 i tried your code to convert TROPOMISIF netcdf to tiff format but i wasnt successful due to the variables in group (PRODUCT). Is there a way i could do that

[pratiman-91]

@pratiman-91 i tried your code to convert TROPOMISIF netcdf to tiff format, but i wasn't successful due to the variables in group (PRODUCT). Is there a way i could do that

hi @JuliusBamah, I am not aware of the file structure of TROPOMISIF. If the netcdf file contains lat and lon, it should work. If netcdf has multiple variables, then you should flconvert GeoTIFF separately. I hope this helps!

[Khan2025]

Hi, I have been trying to convert GRACE/GRAC-FO dataset from NetCDF to GeoTIFF format using your code but it doesn't work. Any help is appreciated.

path = '/content/JPL_MSCNv03_PLACEMENT.nc/' # Change here for file in globe (/content/JPL_MSCNv03_PLACEMENT.nc + '*.nc'): print("Processing: " + file) ncfile = xr.open_dataset(file)

#Extract the variable
pr = ncfile['pr']

#(Optional) convert longitude from (0-360) to (-180 to 180) (if required)
pr.coords['lon'] = (pr.coords['lon'] + 180) % 360 - 180
pr = pr.sortby(pr.lon)

#Define lat/long 
pr = pr.rio.set_spatial_dims('lon', 'lat')

#Check for the CRS
pr.rio.crs

#(Optional) If your CRS is not discovered, you should be able to add it like so:
pr.rio.set_crs("epsg:4326")

#Saving the file
pr.rio.to_raster(file[:-2] + '.tif')

[Khan2025]

Hello, I tried using the code provided in the Google Colab link that I found on Research Gate page. https://colab.research.google.com/drive/1dzAzUKYNvA48RiHNUjPuD9vqO8vy6_7b?usp=sharing While the code worked without any errors, it only returned a single TIFF file, despite the fact that I was expecting monthly files, considering the missing months in the GRACE dataset.

I would greatly appreciate any help you can provide on this matter.

Thank you in advance for your time and consideration.

Best regards, Khan

[pratiman-91]

@Khan2025 I think the code is working just fine. It is expected to return only one single TIFF file with bands equal to 'time'. I am not sure what do you want.