opendatacube / datacube-core

Open Data Cube analyses continental scale Earth Observation data through time
http://www.opendatacube.org
Apache License 2.0
513 stars 178 forks source link

Extend masking functionality with multi-values #552

Open Kirill888 opened 6 years ago

Kirill888 commented 6 years ago

Masking currently supports enumeration types

For example, given flags definition like this:

      flags_definition:
        fmask:
          bits: [0,1,2]
          values:
            '0': unclassified
            '1': valid
            '2': cloud
            '3': shadow
            '4': snow
            '5': water

we can then ask datacube to turn that into a mask this way:

from datacube.storage.masking import make_mask
data = dc.load(..., measurements=['fmask', ...])
m = make_mask(data.fmask, fmask='cloud')

For every pixel that was classified as cloud, m will have True value. But if I wanted to get pixels that were classified as either cloud or shadow I'd have to call make_mask twice

m = make_mask(data.fmask, fmask='cloud') + make_mask(data.fmask, fmask='shadow')

I propose we support following syntax as well:

m = make_mask(data.fmask, fmask=('cloud', 'shadow'))

Not only this adds convenience to the user it also has a potential for reducing peak memory requirement as computation can happen per time-slice.

Implementation Notes

Underlying function create_mask_value can not support multiple values for enumeration, because it assumes that conversion to bool can be represented as X_bool = ((X_raw & M) == V), where M and V are simple integer types and are constructed from a bunch of field_name = field_value_as_a_string pairs.

Combining multiple enum types in one "query" might be a bit tricky.

Kirill888 commented 5 years ago

If the largest value of the enumeration is <= 8,16,32,64 then enumeration can be converted to bit-field mask of type uint{8,16,32,64} simply by doing 1<<v for loaded mask image v. We should consider implementing this case.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

omad commented 1 year ago

There is code inside opendatacube/datacube-ows: Open Data Cube Open Web Services which handles this and should be ported into Core.