Cropland: Malawi 2020

[ivanzvonkov]

Start year: 2020 Start month: September

• [x] Labeling project created
• [x] Labeling completed
• [x] Data added to repository
• [x] Model trained (#294)
• [x] Map made

[ivanzvonkov]

Malawi dashboard: https://hkerner-umd.users.earthengine.app/view/malawi-cropland-in-2020-v1

[MsPixels]

Summary about the Malawi map quality from the call (09/27/2022)

• The terms 'crop type' and 'cropland' were clearly defined
• Incorporate shapefiles and other credible data sources that can improve the overall accuracy of the cropland maps

Next steps

• Use GEE for corrective data labeling for the cropland mapping -Christina and co to discuss the crop type/land use cases

[ivanzvonkov]

Metrics: https://github.com/nasaharvest/crop-mask/blob/7490e2cb090c0a634c6419d9c5fe3eba81cc6701/data/models.json#L104

[ivanzvonkov]

Related: https://github.com/nasaharvest/crop-mask/issues/222

[ivanzvonkov]

Got points from Blake will wait for #251 to finish

[MsPixels]

So far, I have received 28 corrective label files from Ivan. Tried adding the new labels but the Pull Request failed because of the unresolved billing issue

[MsPixels]

Malawi Corrective Labelling Campaign

[ivanzvonkov]

This is an excellent report @MsPixels!! Thank you for doing this analysis. Did you share it with Christina?

[MsPixels]

Thanks Ivan. Yep, I shared this with Christina

[MsPixels]

@ivanzvonkov, this is the error I get when I run the model

[ivanzvonkov]

The Malawi model performs worse with all the new data than when only using high quality data (2 or more labelers). See comparison here: https://api.wandb.ai/links/nasa-harvest/8b6sizbh

This indicates that datasets such as MalawiCorrectiveLabels2020 may still contain incorrect labels that confuse the model and makes #164 higher priority.

Validation metrics are similar to what they were before corrective labeling so I don't think making another map is necessary until there are better metrics. I am thinking of moving this issue to Paused status and reassessing priority on Monday. @MsPixels @hannah-rae let me know what you think.

[ivanzvonkov]

On second thought, the test set metrics of the Malawi model trained on high quality data are higher than I've seen before (higher than validation set even), so maybe we should make a map: https://github.com/nasaharvest/crop-mask/blob/c58a4426d5583d87eb48ab807c4a115437c9bb86/data/models.json#L70

[MsPixels]

After making the map, I got these results - Comparing V1 and V2. There's definitely some changes

[ivanzvonkov]

Potentially do post processing (E.g. filtering out fallow fields re Ben's work, sieve for cleaning noisy predictions [window])

[MsPixels]

Visualizing disagreement layer (V1 vrs V2)