Improve labeling process

[davidgasquez]

A few ideas we can implement.

• Add a way to label specific examples or force relabeling on old examples.
• The person labeling should have the same context the model will have at inference. We should also filter (remove only mentions, replace mentions with @user, ...) and preprocess the data.
• Sort by model uncertainty. We should focus on False Positives and labels without much confidence. Similar to what Prodigy does.

[davidgasquez]

These are some great apps for labeling text:

• https://github.com/heartexlabs/label-studio. You can add a backend to label and use Active Learning to help the process.
• https://github.com/doccano/doccano.
• https://github.com/UniversalDataTool/universal-data-tool

[davidgasquez]

Having some issues setting up LabeL Studio with persistent storage. Reached out to them via Github: https://github.com/heartexlabs/label-studio/issues/516.