terraref / reference-data

Coordination of Data Products and Standards for TERRA reference data
https://terraref.org
BSD 3-Clause "New" or "Revised" License
9 stars 2 forks source link

Canopy cover data should handle missing data #238

Open dlebauer opened 6 years ago

dlebauer commented 6 years ago

Currently the canopy cover extractor runs on full field stitched images. However, it does not correctly handle missing data. Thus, on days with partial coverage the missing data weights the means toward 0, and we end up with plots in which the value of canopy cover is erratic, for example:

image

It doesn't appear that the solution will be as simple as deleting zeros after emergence, since some plots may have partial coverage.

The new image to plot aggregation will solve this issue, but in the mean time we need to remove data that is in error from the database.

@ZongyangLi is there a way to identify (and thus remove) data that is invalid without re-processing?

TODO

dlebauer commented 6 years ago

After discussing with @max-zilla today, it sounds like the canopy cover extractor can be re-run with existing updates that allow it to handle missing values.

dlebauer commented 5 years ago

Purged all previous canopy_cover data where checked > -1

delete from traits where  
    traits.variable_id in (select id from variables where name = 'canopy_cover') and 
    method_id in (select id from methods where name = 'Canopy Cover Estimation from Field Scanner RGB images') and 
        checked > -1;