Calibration fails for the Sentinel-1 SAFE format

[Polarnix]

Hi,

I'm having trouble exporting Sentinel-1 images. Previously something like the following would work:

from nansat import Nansat
inpfn = 'S1A_EW_GRDM_1SDH_20180823T082256_20180823T082320_023371_028AE4_856D.SAFE'
s1i = Nansat(inpfn,mapperName='sentinel1_l1')
outbands = [ 'sigma0_HH' ]
s1i.export('test_output.tif',bands=outbands,driver='GTiff',options=['BIGTIFF=YES'])

It appears only the sigma0 and beta0 channels are affected, so I suspect the Sentinel1Calibration pixel function and interaction with the GDAL CreateCopy function. With GDAL 2.2.3 this just throws an uninformative segmentation fault, but the latest GDAL (2.3.1) reports the following

Warning 1: Invalid band number. Got 8, expected 7. Ignoring provided one, and using 7 instead Warning 1: Invalid band number. Got 8, expected 6. Ignoring provided one, and using 6 instead Warning 1: Invalid band number. Got 8, expected 5. Ignoring provided one, and using 5 instead Warning 1: Invalid band number. Got 8, expected 4. Ignoring provided one, and using 4 instead Warning 1: Invalid band number. Got 8, expected 3. Ignoring provided one, and using 3 instead Warning 1: Invalid band number. Got 8, expected 2. Ignoring provided one, and using 2 instead Warning 1: Invalid band number. Got 8, expected 1. Ignoring provided one, and using 1 instead Traceback (most recent call last): File "test_nansat_iw.py", line 44, in <module> s1i.export(tiffn,bands=outbands,driver='GTiff',options=['BIGTIFF=YES']) File "/home/nicholsh/Nansat/lib/python2.7/site-packages/nansat-1.1.3-py2.7-linux-x86_64.egg/nansat/exporter.py", line 116, in export dataset = gdal.GetDriverByName(driver).CreateCopy(filename, export_vrt.dataset, options=options) File "/usr/local/lib/python2.7/dist-packages/osgeo/gdal.py", line 1538, in CreateCopy return _gdal.Driver_CreateCopy(self, *args, **kwargs) RuntimeError: Too many threads are running for the current value of the dataset pool size (100). or too many proxy datasets are opened in a cascaded way. Try increasing GDAL_MAX_DATASET_POOL_SIZE.

If I try NetCDF output instead of GeoTIFF, I don't get the traceback report but the file created does not contain any valid pixels.

Cheers,

Nick

[fabricebrito]

+1

os.environ['GDAL_MAX_DATASET_POOL_SIZE'] = '450' won't help

[mortenwh]

The failing export seems to be related to an issue with the calibration. GDAL needs to read the band data before exporting. I've tried with this dataset: S1A_IW_GRDH_1SDV_20190617T051908_20190617T051937_027715_0320DB_A8A7.SAFE

In [6]: n['sigma0_VV']                                                                                             
---------------------------------------------------------------------------
NansatGDALError                           Traceback (most recent call last)
<ipython-input-6-a552f9008a1e> in <module>
----> 1 n['sigma0_VV']

~/Miniconda3-4.6.14-Linux-x86_64/envs/py3openwind/lib/python3.6/site-packages/nansat/nansat.py in __getitem__(self, band_id)
    183         band_data = band.ReadAsArray()
    184         if band_data is None:
--> 185             raise NansatGDALError('Cannot read array from band %s' % str(band_data))
    186 
    187         # execute expression if any

NansatGDALError: Cannot read array from band None

This is the output when trying to export:

In [4]: n.export('test_output.tif',bands=outbands,driver='GTiff',options=['BIGTIFF=YES'])                          
Warning 1: Invalid band number. Got 10, expected 9. Ignoring provided one, and using 9 instead
Warning 1: Invalid band number. Got 10, expected 8. Ignoring provided one, and using 8 instead
Warning 1: Invalid band number. Got 10, expected 7. Ignoring provided one, and using 7 instead
Warning 1: Invalid band number. Got 10, expected 6. Ignoring provided one, and using 6 instead
Warning 1: Invalid band number. Got 10, expected 5. Ignoring provided one, and using 5 instead
Warning 1: Invalid band number. Got 10, expected 4. Ignoring provided one, and using 4 instead
Warning 1: Invalid band number. Got 10, expected 3. Ignoring provided one, and using 3 instead
Warning 1: Invalid band number. Got 10, expected 2. Ignoring provided one, and using 2 instead
Warning 1: Invalid band number. Got 10, expected 1. Ignoring provided one, and using 1 instead
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-45db113737ec> in <module>
----> 1 n.export('test_output.tif',bands=outbands,driver='GTiff',options=['BIGTIFF=YES'])

~/Miniconda3-4.6.14-Linux-x86_64/envs/py3openwind/lib/python3.6/site-packages/nansat/exporter.py in export(self, filename, bands, rm_metadata, add_geolocation, driver, options, hardcopy)
    116 
    117         # Create output file using GDAL
--> 118         dataset = gdal.GetDriverByName(driver).CreateCopy(filename, export_vrt.dataset, options=options)
    119         del dataset
    120         # add GCPs into netCDF file as separate float variables

~/Miniconda3-4.6.14-Linux-x86_64/envs/py3openwind/lib/python3.6/site-packages/osgeo/gdal.py in CreateCopy(self, *args, **kwargs)
   1746     def CreateCopy(self, *args, **kwargs):
   1747         """CreateCopy(Driver self, char const * utf8_path, Dataset src, int strict=1, char ** options=None, GDALProgressFunc callback=0, void * callback_data=None) -> Dataset"""
-> 1748         return _gdal.Driver_CreateCopy(self, *args, **kwargs)
   1749 
   1750 

RuntimeError: /vsimem/2FSMG7HNFA.vrt: GDALDataset::GetRasterBand(2) - Illegal band #

[georgbachmann]

I am seeing the error of

Too many threads are running for the current value of the dataset pool size (100). or too many proxy datasets are opened in a cascaded way. Try increasing GDAL_MAX_DATASET_POOL_SIZE

myself in a different project (tilemill) but googling the error brought me here. I was just wondering if/how you solved the probleme here? Setting it as an environment variable did not work either for me. Cheers, Georg