Add notebook for CEO meta analysis

[bhyeh]

Notebook for generating simple meta insights to CEO labeling project.

Remaining items and thoughts:

• At the moment - a user needs to manually inspect CEO csv files to determine the date when plot labeling is completed by both sets. This process could be automated at some point too.
• Add more documentation about where user should dump csv files and specifying path location in nb.
• Q. 2.3.2 Which overridden labels have the highest analysis duration? - Add the distribution of incorrect and mistaken labels before label-label breakdown.

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[bhyeh]

2/20/23 meeting notes

1. For crop land map and crop type map - gather validation and test set via CEO and two labeler sets a. For doing just crop land/type mapping - disagreements are 'thrown out' and ignored inputs to model b. For additionally doing area estimation - disagreements are resolved/forced into agreement because 'throwing out' or ignoring these points affect the sample size $\rightarrow$ downstream effect for map accuracy assessment (e.g., width of confidence interval).
2. Reasons for meta analysis on disagreement points a. For just crop land/type mapping - looking at patterns of disagreements help to build a systematic resolvement or protocol instead of just ignoring #275 b. For area estimation - help identify ambiguous points to be used towards 'counterfactual' or alternative accuracy assessment

Notes:

1. Most typical is crop land/type map project $\rightarrow$ where result of CEO labeling is two csv files from labeler sets. There is no 'final' label known (b/c disagreement points are just ignored, not forced into agreement, as opposed to area estimate label project).

[ivanzvonkov]

FYI pre-commit is now working and there's a couple issues it highlights (see pre-commit.ci "Details" link)

[ivanzvonkov]

@bhyeh let me know when this is ready for a second look!

[ivanzvonkov]

flake8 has some issues with the consensus_utils file, you can see what they are by clicking on the pre-commit.ci "Details"

[ivanzvonkov]

This is looking pretty good, any thoughts about some of the notebook comments? I am happy to merge soon and iterate if need be.

[bhyeh]

This is looking pretty good, any thoughts about some of the notebook comments? I am happy to merge soon and iterate if need be.

Thanks! Yes I do plan on making further changes on the notebook - waiting for some feedback from Hannah and will then address your comments there as well.

[ivanzvonkov]

@bhyeh can this be merged in the near term (after pre-commit issues are addressed)?

[bhyeh]

@bhyeh can this be merged in the near term (after pre-commit issues are addressed)?

I think I need to touch base w/ @hannah-rae for her thoughts. I'll bring this up again with our 1:1 this Monday.