search

ubarsc / rios

A raster processing layer on top of GDAL
https://www.rioshome.org
GNU General Public License v3.0
15 stars 8 forks source link

calcstats.addStatistics can fail when the band contains no valid pixels #103

Closed tonykgill closed 4 months ago

tonykgill commented 4 months ago

Yes, this is an odd case. I am using rios's addStatistics function to calculate stats for images created outside of rios.

When a band contains no valid pixels, that is all pixels are equal to the band's no-data value, calcstats.addStatistics fails for images with data type uint16 (I've only tested with images of type byte and uint16. byte images do not raise this exception).

Traceback (most recent call last):
  ...
  ...
    calcstats.addStatistics(ds_tif, progress, approx_ok=True)
  File "/cibosw/lib/python3.10/site-packages/rios/calcstats.py", line 238, in addStatistics
    histrange = int(numpy.ceil(maxval) - numpy.floor(minval)) + 1
TypeError: must be real number, not NoneType

Using the attached images and script (in the zip file), you may reproduce as follows. rios_issue.zip

$ ./try_calc_stats.py multi_bands_uint16.tif
Error calculating statistics for multi_bands_uint16.tif: must be real number, not NoneType Stack Trace: Traceback (most recent call last):
  File "/home/tony/dev/cibolabs/cibo_agriwebb_lite/rios_stats_bug/./try_calc_stats.py", line 15, in <module>
    calcstats.addStatistics(ds_tif, progress, approx_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/rios/calcstats.py", line 238, in addStatistics
    histrange = int(numpy.ceil(maxval) - numpy.floor(minval)) + 1
TypeError: must be real number, not NoneType

$ ./try_calc_stats.py single_band_uint16.tif 
Error calculating statistics for single_band_uint16.tif: must be real number, not NoneType Stack Trace: Traceback (most recent call last):
  File "/home/tony/dev/cibolabs/cibo_agriwebb_lite/rios_stats_bug/./try_calc_stats.py", line 15, in <module>
    calcstats.addStatistics(ds_tif, progress, approx_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/rios/calcstats.py", line 238, in addStatistics
    histrange = int(numpy.ceil(maxval) - numpy.floor(minval)) + 1
TypeError: must be real number, not NoneType

$ ./try_calc_stats.py three_bands_byte.tif
# No error
neilflood commented 4 months ago

So, I think the problem is with inconsistent use of the addStatistics function. Not really your fault, as this is not obvious. To make this go away, you just need to explicitly pass in the ignore value.

When used within RIOS, it is always given an explicit ignore value. Inside the function, GDAL won't accept this, and uses the value written in the file. When GDAL finds no valid pixels, it raises an exception, which we catch, and we set the maxval to the given ignore value, which in this case is None.

Possibly I should have an extra trap so that if the ignore parameter is None, it should also check the one in the file. I am unsure if this would create other problems, so I won't rush into that. Anyway, just pass the known ignore value in to addStatistics, and it will be OK. Will that be acceptable for now?

tonykgill commented 4 months ago

Will that be acceptable for now?

Yes. Absolutely. Thanks Neil.

neilflood commented 4 months ago

Just a further note. Within RIOS, we are using addStatistics() to actually set the null value, as well as add the stats. Again, this is not obvious, once you start using that routine outside of RIOS. I will ponder the best approach - it may be that just improving the docstring is really the right solution.

tonykgill commented 4 months ago

Yes. I suspect updating the docstring is all that is required.