search

davidfrantz / force

Framework for Operational Radiometric Correction for Environmental monitoring
GNU General Public License v3.0
172 stars 50 forks source link

Masked blocks in TSS are assigned 0 instead of nodata value #234

Closed ernstste closed 1 year ago

ernstste commented 1 year ago

Maybe related to #26:

When producing TSS with masks in v3.7.8, there are some masked areas that are assigned 0 instead of the nodata value. The issue does not occur when using v3.7.7. We also didn't observe this when Landsat was processed for the same 30x30km tile - same block size (3000) but 30m res instead of 10m.

sdf

davidfrantz commented 1 year ago

I wonder what I changed... I guess I will need to implement the CSO fix in each sub-module

davidfrantz commented 1 year ago

Hey @ernstste, did https://github.com/davidfrantz/force/issues/26 ever work? I just made a small test, and it doesn't work for TSA and CSO.

I tried to pin down the issue, and problem is, that data for this chunk are neither read, computed, nor written. I guess that the GeoTiff is sparse, though I do not understand yet why this worked in the past.

davidfrantz commented 1 year ago

Damn

davidfrantz commented 1 year ago

This bug was introduced when implementing the custom GDAL options where you can define your own output format and choose your own driver.

Not every driver supports "creating" a dataset, some need to create by copying an existing one. This can be an in-memory dataset, however. And to be most flexible and simplify the code, I followed that strategy for all drivers, i.e.

1) create dataset in-memory 2) copy dataset to disk

Apparently, this does not work with sparse files somehow... ugh.

I guess I will need to check if the driver can create files, and then work around this by either directly creating or create/copying only if needed.

ernstste commented 1 year ago

Ouch. Custom GDAL options come back to bite you again.

I'm pretty sure #26 did work back then. I tried it right after you implemented the fix and the results looked ok.

davidfrantz commented 1 year ago

It might be, that the problem does not consistently surface - you also mentioned that you did not see that issue with Landsat data.

davidfrantz commented 1 year ago

I worked on a fix: https://github.com/davidfrantz/force/commit/d9759f64204a563723ec39f63970928267a77751

I think it solves this issue.

The only thing to mention (and I believe this already did not work before) is that I could not make create/copy drivers work for block-output (like in HLPS). If full files are written, it works, however (like the OVV images in L2PS).