search

ESA-PhiLab / OpenSarToolkit

High-level functionality for the inventory, download and pre-processing of Sentinel-1 data in the python language.
https://opensartoolkit.readthedocs.io/en/latest/
MIT License
211 stars 54 forks source link

Problem creating ARD GeoTiff, GDAL TypeError "['to_tif'] = True" #87

Open alexisahedo opened 2 years ago

alexisahedo commented 2 years ago

Hello dear everyone,

I would like to report an issue I have been facing trying to create an ARD GeoTiff output from a Sentinel-1 scene.

My objective is to create a GeoTiff output of the ARD CEOS standard hopping to index and ingest it in an Open Data Cube implementation, the problem is that it seems that GDAL is doing something wrong and popping me an error. I would really appreciate if someone would be my lifesaver and help me to solve this problem, which I’m not completely sure if it’s related to my GDAL installation or some other problem. I have been able to create BEAM-DIMAP (.bim) output of the ARG CEOS standard without problem, and even RGB with the just mentioned files.

Terminal error output.

Wed 27/04/22 [10:32:26 AM]: INFO [grd_to_ard.py:57] Processing acquisition from 20180630 over track 114. INFO (10:32:26): Processing acquisition from 20180630 over track 114. Executing processing graph ....10%....20%....30%....40%....50%....60%....70%....80%....90% done. Executing operator... 20%....30%....40%....50%....60%....70%....80%....90%.... done. Writing... .13%..24%....34%.....44%....54%....64%....74%....84%....94%.. done. Executing processing graph ....10%....20%....30%....40%....50%....60%....70%....80%....90% done. Executing operator... 20%....30%....40%....50%....60%....70%....80%....90%.... done. Writing... .13%..24%....34%....44%....54%....64%....74%....84%....94%.. done. Executing operator... 20%....30%....40%....50%....60%....70%....80%....90%.... done. Writing... .13%..24%....34%....44%....54%....64%....74%....84%....94%... done. Traceback (most recent call last): File "Path/ost_prueba_02/ost_Sentinel1Scene_test.py", line 170, in scene_s1.create_ard( File "Path/anaconda3/envs/venv_ost/lib/python3.8/site-packages/ost/s1/s1scene.py", line 858, in create_ard filelist, out_bs, out_ls, error = grd_to_ard( File "Path/anaconda3/envs/venv_ost/lib/python3.8/site-packages/ost/s1/grd_to_ard.py", line 408, in grd_to_ard h.move_dimap(geocoded, out_final, ard["to_tif"]) File "Path/anaconda3/envs/venv_ost/lib/python3.8/site-packages/ost/helpers/helpers.py", line 105, in move_dimap gdal.Warp(outfile_prefix.with_suffix(".tif"), infile_prefix.with_suffix(".dim")) File "Path/anaconda3/envs/venv_ost/lib/python3.8/site-packages/osgeo/gdal.py", line 709, in Warp return wrapper_GDALWarpDestDS(destNameOrDestDS, srcDSTab, opts, callback, callback_data) File "Path/anaconda3/envs/venv_ost/lib/python3.8/site-packages/osgeo/gdal.py", line 4703, in wrapper_GDALWarpDestDS return _gdal.wrapper_GDALWarpDestDS(args) TypeError: in method 'wrapper_GDALWarpDestDS', argument 1 of type 'GDALDatasetShadow '

Configuration (processing.json):

{
    "processing": {
        "single_ARD": {
            "image_type": "GRD",
            "ard_type": "CEOS",
            "resolution": 10,
            "remove_border_noise": true,
            "product_type": "RTC-gamma0",
            "polarisation": "VV, VH, HH, HV",
            "to_db": false,
            "to_tif": true,
            "geocoding": "terrain",
            "remove_speckle": false,
            "speckle_filter": {
                "filter": "Refined Lee",
                "ENL": 1,
                "estimate_ENL": true,
                "sigma": 0.9,
                "filter_x_size": 3,
                "filter_y_size": 3,
                "window_size": "7x7",
                "target_window_size": "3x3",
                "num_of_looks": 1,
                "damping": 2,
                "pan_size": 50
            },
            "create_ls_mask": true,
            "apply_ls_mask": false,
            "dem": {
                "dem_name": "Copernicus 30m Global DEM",
                "dem_file": "",
                "dem_nodata": 0,
                "dem_resampling": "BILINEAR_INTERPOLATION",
                "image_resampling": "BICUBIC_INTERPOLATION",
                "egm_correction": false,
                "out_projection": 4326
            }
        },
        "time-series_ARD": {
            "to_db": true,
            "remove_mt_speckle": false,
            "apply_ls_mask": false,
            "mt_speckle_filter": {
                "filter": "Refined Lee",
                "ENL": 1,
                "estimate_ENL": true,
                "sigma": 0.9,
                "filter_x_size": 3,
                "filter_y_size": 3,
                "window_size": "7x7",
                "target_window_size": "3x3",
                "num_of_looks": 1,
                "damping": 2,
                "pan_size": 50
            },
            "deseasonalize": false,
            "dtype_output": "float32"
        },
        "time-scan_ARD": {
            "metrics": [
                "avg",
                "max",
                "min",
                "std",
                "cov"
            ],
            "remove_outliers": false,
            "apply_ls_mask": false
        },
        "mosaic": {
            "harmonization": true,
            "production": false,
            "cut_to_aoi": false
        }
    },
    "processing_dir": "/disco03/adiaz/Documentos/api_pruebas_01/ost_prueba_02/P01/processing/Geotiff",
    "temp_dir": "/disco03/adiaz/Documentos/api_pruebas_01/ost_prueba_02/P01/processing/Geotiff/temp",
    "snap_cpu_parallelism": 80,
    "subset": false
}

Virtual environment libraries list (Anaconda, python 3.8): 

Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
attrs                     21.4.0                   pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
blosc                     1.21.0               h9c3ff4c_0    conda-forge
boost-cpp                 1.74.0               h6cacc03_7    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cairo                     1.16.0            ha12eb4b_1010    conda-forge
certifi                   2021.10.8        py38h578d9bd_2    conda-forge
cfitsio                   4.1.0                hd9d235c_0    conda-forge
charset-normalizer        2.0.12                   pypi_0    pypi
click                     8.1.2                    pypi_0    pypi
curl                      7.82.0               h2283fc2_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
expat                     2.4.8                h27087fc_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.32.0                   pypi_0    pypi
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
gdal                      3.4.2            py38h51ade5a_5    conda-forge
geos                      3.10.2               h9c3ff4c_0    conda-forge
geotiff                   1.7.1                h509b78c_1    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
hdf4                      4.2.15               h10796ff_3    conda-forge
hdf5                      1.12.1          nompi_h4df4325_104    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.3                      pypi_0    pypi
importlib-resources       5.7.1                    pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
jbig                      2.1               h7f98852_2003    conda-forge
jinja2                    3.1.1                    pypi_0    pypi
jpeg                      9e                   h166bdaf_1    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
jsonschema                4.4.0                    pypi_0    pypi
jupyter-core              4.10.0                   pypi_0    pypi
kealib                    1.4.14               h87e4c3c_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.2                    pypi_0    pypi
krb5                      1.19.3               h08a2579_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libcurl                   7.82.0               h2283fc2_0    conda-forge
libdap4                   3.20.6               hd7c4107_2    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_15    conda-forge
libgdal                   3.4.2                h5750acb_5    conda-forge
libgfortran-ng            11.2.0              h69a702a_15    conda-forge
libgfortran5              11.2.0              h5c6108e_15    conda-forge
libglib                   2.70.2               h174f98d_4    conda-forge
libgomp                   11.2.0              h1d223b6_15    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h238a007_1014    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnetcdf                 4.8.1           nompi_hb3fd0d9_101    conda-forge
libnghttp2                1.47.0               he49606f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     14.2                 h676c864_0    conda-forge
librttopo                 1.1.0                hf69c175_9    conda-forge
libspatialite             5.0.1               ha867d66_15    conda-forge
libssh2                   1.10.0               ha35d2d1_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_15    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzip                    1.8.0                h1c5bbd1_1    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
matplotlib                3.5.1                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.77                 h2350873_0    conda-forge
numpy                     1.22.3           py38h1d589f8_2    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
opensartoolkit            0.12.15                  pypi_0    pypi
openssl                   3.0.2                h166bdaf_1    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0                    pypi_0    pypi
pip                       22.0.4                   pypi_0    pypi
pixman                    0.40.0               h36c2ea0_0    conda-forge
poppler                   22.01.0              h1434ded_2    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                14.2                 hce44dc1_0    conda-forge
proj                      9.0.0                h93bde94_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyparsing                 3.0.8                    pypi_0    pypi
pyproj                    3.3.0                    pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
python                    3.8.13          ha86cf86_0_cpython    conda-forge
python_abi                3.8                      2_cp38    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1                   pypi_0    pypi
retrying                  1.3.3                    pypi_0    pypi
setuptools                62.1.0           py38h578d9bd_0    conda-forge
six                       1.16.0                   pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
sqlite                    3.38.2               h4ff8645_0    conda-forge
tiledb                    2.7.2                h3f4058f_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzcode                    2022a                h166bdaf_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
urllib3                   1.26.9                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.3                h8ce2273_4    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge

I apologize for the huge text; I hope it will be of help in identifying what the problem is. Thanks a lot for all your help, I really appreciate it.

Alexis A.

BuddyVolly commented 2 years ago

I'll check if we can move to rasterio for this and make it work again. gdal is a headache, and so is snap...bringing both together as in this case is just double trouble. I'll keep you updated. BV

alexisahedo commented 2 years ago

Dear BuddyVolly,

Thanks a lot for your answer, as always, I really appreciate all your help. I really do understand the big headache this is, I’ll patiently wait for your updates, my biggest cheers to you.

Saludos. Alexis K.