too large image?

[aleksandervines]

I wanted to find out the memory requirements for the sentinel1ice program. But when I opened a huge image - S1A_EW_GRDH_1SDH_20180321T072153_20180321T072403_021110_024456_25A0.zip which is 2.2GB, I got a RuntimeError in S1_EW_GRD_NoiseCorrection

I will look a bit further into this. But has anyone seen this before?

12:58:17|40|nansat|_get_dataset_metadata|GDAL could not open /Home/alevin/nextgeos/test/1/S1A_EW_GRDH_1SDH_20180321T072153_20180321T072403_021110_024456_25A0.zip, trying to read with Nansat mappers...
denoising and multi-look ...
Traceback (most recent call last):
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/mnt/10.11.12.231/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/memory_profiler.py", line 1228, in <module>
    exec_with_profiler(script_filename, prof, args.backend, script_args)
  File "/mnt/10.11.12.231/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/memory_profiler.py", line 1123, in exec_with_profiler
    execfile(filename, ns, ns)
  File "07_apply_svm.py", line 42, in <module>
    sigma0, tfs, pca_zones, svm_zones = get_map(s1i,env, ia)
  File "sar2ice.py", line 452, in get_map
    angDepCor=True, fillVoid=False, dBconv=False, development=True )
  File "/Home/alevin/nextgeos/sentinel1_class/sentinel1denoised/S1_EW_GRD_NoiseCorrection.py", line 645, in add_denoised_band
    denoisedBandArray = self.get_denoised_band(bandName, **kwargs)
  File "/Home/alevin/nextgeos/sentinel1_class/sentinel1denoised/S1_EW_GRD_NoiseCorrection.py", line 880, in get_denoised_band
    parameters={'name': 'NEsigma0_'+pol+'_raw'})
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/nansat/nansat.py", line 330, in add_band
    self.add_bands([array], [parameters], nomem)
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/nansat/nansat.py", line 365, in add_bands
    vrt = VRT.from_array(array, nomem=nomem)
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/nansat/vrt.py", line 218, in from_array
    vrt._init_from_array(array, **kwargs)
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/nansat/vrt.py", line 391, in _init_from_array
    gdal.VSIFWriteL(array.tostring(), len(array.tostring()), 1, ofile)
  File "/Home/alevin/miniconda2/envs/TEST/lib/python2.7/site-packages/osgeo/gdal.py", line 1434, in VSIFWriteL
    return _gdal.VSIFWriteL(*args)
RuntimeError: too large buffer (>2GB)

[akorosov]

I didn't see that before

[Polarnix]

Hi,

Now that IW mode is referred to in the routine, I see this occurring with IW images:

Traceback (most recent call last): File "/home/nicholsh/git/s1-gridengine/Python/test_nansat.py", line 32, in <module> s1i.add_denoised_band('HH') File "/home/nicholsh/git/sentinel1denoised/S1_TOPS_GRD_NoiseCorrection.py", line 1050, in add_denoised_band parameters={'name': 'sigma0_%s' % polarization + '_raw'},nomem=True) File "/home/nicholsh/Nansat/lib/python2.7/site-packages/nansat-1.1.3-py2.7-linux-x86_64.egg/nansat/nansat.py", line 295, in add_band self.add_bands([array], [parameters], nomem) File "/home/nicholsh/Nansat/lib/python2.7/site-packages/nansat-1.1.3-py2.7-linux-x86_64.egg/nansat/nansat.py", line 330, in add_bands vrt = VRT.from_array(array, nomem=nomem) File "/home/nicholsh/Nansat/lib/python2.7/site-packages/nansat-1.1.3-py2.7-linux-x86_64.egg/nansat/vrt.py", line 202, in from_array vrt._init_from_array(array, **kwargs) File "/home/nicholsh/Nansat/lib/python2.7/site-packages/nansat-1.1.3-py2.7-linux-x86_64.egg/nansat/vrt.py", line 376, in _init_from_array gdal.VSIFWriteL(array.tostring(), len(array.tostring()), 1, ofile) File "/usr/local/lib/python2.7/dist-packages/osgeo/gdal.py", line 1434, in VSIFWriteL return _gdal.VSIFWriteL(*args) RuntimeError: too large buffer (>2GB)

I tried adding the nomem=True to the relevant lines in S1_Tops_GRD_NoiseCorrection.py but that didn't have any effect.

Cheers,

Nick