Entity linking

[percevalw]

Preview docs: https://aphp.github.io/edsnlp/dev

Description

Added

• Support for a filesystem parameter in every edsnlp.data.read_* and edsnlp.data.write_* functions
• Pipes of a pipeline are now easily accessible with nlp.pipes.xxx instead of nlp.get_pipe("xxx")

• Support builtin Span attributes in converters span_attributes parameter, e.g.

  import edsnlp
  
  nlp = ...
  nlp.add_pipe("eds.sentences")
  
  data = edsnlp.data.from_xxx(...)
  data = data.map_pipeline(nlp)
  data.to_pandas(converters={"ents": {"span_attributes": ["sent.text", "start", "end"]}})

• Support assigning Brat AnnotatorNotes as span attributes: edsnlp.data.read_standoff(..., notes_as_span_attribute="cui")

• Support for mapping full batches in edsnlp.processing pipelines with map_batches lazy collection method:

  import edsnlp
  
  data = edsnlp.data.from_xxx(...)
  data = data.map_batches(lambda batch: do_something(batch))
  data.to_pandas()

• New data.map_gpu method to map a deep learning operation on some data and take advantage of edsnlp multi-gpu inference capabilities
• Added average precision computation in edsnlp span_classification scorer

• You can now add pipes to your pipeline by instantiating them directly, which comes with many advantages, such as auto-completion, introspection and type checking !

  import edsnlp, edsnlp.pipes as eds
  
  nlp = edsnlp.blank("eds")
  nlp.add_pipe(eds.sentences())
  # instead of nlp.add_pipe("eds.sentences")

  The previous way of adding pipes is still supported.

Changed

• nlp.preprocess_many now uses lazy collections to enable parallel processing
• :warning: Breaking change. Improved and simplified eds.span_qualifier: we didn't support combination groups before, so this feature was scrapped for now. We now also support splitting values of a single qualifier between different span labels.
• Optimized edsnlp.data batching, especially for large batch sizes (removed a quadratic loop)
• :warning: Breaking change. By default, the name of components added to a pipeline is now the default name defined in their class __init__ signature. For most components of EDS-NLP, this will change the name from "eds.xxx" to "xxx".

Fixed

• Flatten list outputs (such as "ents" converter) when iterating: nlp.map(data).to_iterable("ents") is now a list of entities, and not a list of lists of entities
• Allow span pooler to choose between multiple base embedding spans (as likely produced by eds.transformer) by sorting them by Dice overlap score.
• EDS-NLP does not raise an error anymore when saving a model to an already existing, but empty directory

Checklist

• [x] If this PR is a bug fix, the bug is documented in the test suite.
• [x] Changes were documented in the changelog (pending section).
• [x] If necessary, changes were made to the documentation (eg new pipeline).

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 99.75490% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 97.43%. Comparing base (b4b951a) to head (00fdd59).

Files	Patch %	Lines
edsnlp/core/torch_component.py	90.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #280 +/- ## ========================================== + Coverage 97.29% 97.43% +0.14% ========================================== Files 257 262 +5 Lines 8753 9090 +337 ========================================== + Hits 8516 8857 +341 + Misses 237 233 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.