Is your feature request related to a problem? Please describe.
We're processing AVHRR data. AVHRR may switch between channels 3A and 3B in the middle of a swath. This means both channels 3A and 3B and their composites can be loaded. After resampling to the area of interest, up to 100% of the resampled scene may be invalid (NaN) for either of those channels and composites using them. Pytroll will write images containing no valid data. In our client software (NinJo), these images will be shown as available, but when loading, nothing will be shown (because everything is fill value), and the user will think something is wrong. I'd like that trollflow2 can skip products that are mostly or entirely fill value.
Describe the solution you'd like
I'd like to add a plugin check_valid or something like that, that verifies that at least x% (configurable) of a product (channel or composite) is valid (either by .notnull() or by .notnull()/covers()). If not, this product is removed (similar to sunlight coverage checks).
Describe any changes to existing user workflow
Fully backward compatible.
Additional context
I could remove files that are suspiciously small, but this kind of check should be done before, not after, saving datasets.
Feature Request
Is your feature request related to a problem? Please describe.
We're processing AVHRR data. AVHRR may switch between channels 3A and 3B in the middle of a swath. This means both channels 3A and 3B and their composites can be loaded. After resampling to the area of interest, up to 100% of the resampled scene may be invalid (NaN) for either of those channels and composites using them. Pytroll will write images containing no valid data. In our client software (NinJo), these images will be shown as available, but when loading, nothing will be shown (because everything is fill value), and the user will think something is wrong. I'd like that trollflow2 can skip products that are mostly or entirely fill value.
Describe the solution you'd like
I'd like to add a plugin
check_validor something like that, that verifies that at least x% (configurable) of a product (channel or composite) is valid (either by.notnull()or by.notnull()/covers()). If not, this product is removed (similar to sunlight coverage checks).Describe any changes to existing user workflow
Fully backward compatible.
Additional context
I could remove files that are suspiciously small, but this kind of check should be done before, not after, saving datasets.