Reference analysis

[adamjanovsky]

This PR adds models to annotate references.

Things to resolve:

• [x] This reverts from billiard to multiprocessing, check that we're ok with that
• [x] Label test examples for annotations -- otherwise the annotator will not work in production
• [x] Add analysis of references
• [x] If we are to further process the references, consider storing the labels in Enum. Right now they're just string
• [x] Test that the annotator works in production.
• [x] Decide how to approach nlp dependencies. We either need to add new extra [nlp] and guards or keep them in ordinary requirements.
• [x] Maybe write some tests? But that's going to be painful
• [ ] Actually call annotate_references() from somewhere. Probably from analyze_certificates()?

As a byproduct, this closes #256

[codecov[bot]]

Codecov Report

Attention: 675 lines in your changes are missing coverage. Please review.

Comparison is base (f0f7fa7) 74.61% compared to head (68b1c2d) 67.80%.

:exclamation: Current head 68b1c2d differs from pull request most recent head b648241. Consider uploading reports for the commit b648241 to get more accurate results

Files	Patch %	Lines
...c_certs/model/references_nlp/feature_extraction.py	0.00%	266 Missing :warning:
...ec_certs/model/references_nlp/segment_extractor.py	0.00%	154 Missing :warning:
...ec_certs/model/references_nlp/annotator_trainer.py	0.00%	80 Missing :warning:
src/sec_certs/model/references_nlp/annotator.py	0.00%	56 Missing :warning:
src/sec_certs/model/references_nlp/evaluation.py	0.00%	50 Missing :warning:
src/sec_certs/model/references_nlp/training.py	0.00%	37 Missing :warning:
src/sec_certs/utils/nlp.py	0.00%	20 Missing :warning:
src/sec_certs/model/references_nlp/__init__.py	0.00%	10 Missing :warning:
src/sec_certs/dataset/cc.py	75.00%	1 Missing :warning:
src/sec_certs/sample/cc_certificate_id.py	85.72%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #322 +/- ## ========================================== - Coverage 74.61% 67.80% -6.80% ========================================== Files 54 62 +8 Lines 6709 7394 +685 ========================================== + Hits 5005 5013 +8 - Misses 1704 2381 +677 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[adamjanovsky]

Just a quick comment on the current state of affairs:

• Soon, hyperparameter tunning will be done. Scripts for hyperparam tuning will not be versioned, I'll back them up separately at GDrive
• I don't think that the annotation engine is production-ready. For that reason, I kept the necessary files in sec_certs.model.references_nlp package that has some complex dependencies. Any import from that package requires sec-certs installation with [nlp] extra. A check is done on any import from that package.
• This requires that all nlp deps are also installed on dev machines -- otherwise, linters will likely complain
• In summary, this PR adds:
  • Backend for Reference annotations
  • Notebooks with reference annotation analysis
  • Versioned manual annotations, scripts to measure inter-annotator agreement