Data connectors and processing - Githubissues

aphp / edsnlp

Modular, fast NLP framework, compatible with Pytorch and spaCy, offering tailored support for French clinical notes.

https://aphp.github.io/edsnlp/

BSD 3-Clause "New" or "Revised" License

112 stars 29 forks source link

Data connectors and processing #224

Closed percevalw closed 9 months ago

percevalw commented 9 months ago

Description

Added

New add unified edsnlp.data api (json, brat, spark, pandas) and LazyCollection object to efficiently read / write data from / to different formats & sources.
New unified processing API to select the execution execution backends via docs.configure(...)
The training scripts can now use data from multiple concatenated adapters
Support quantized transformers (compatible with multiprocessing as well !)

Changed

Pipes (in edsnlp/pipelines) are now lazily loaded, which should improve the loading time of the library.
to_disk methods can now return a config to override the initial config of the pipeline (e.g., to load a transformer directly from the path storing its fine-tuned weights)
The eds.tokenizer tokenizer has been added to entry points, making it accessible from the outside
Deprecate old connectors (e.g. BratDataConnector) in favor of the new edsnlp.data API
Deprecate old pipe wrapper in favor of the new processing API

Fixed

Support for pydantic v2
Support for python 3.11 (not ci-tested yet)
Checklist

[x] If this PR is a bug fix, the bug is documented in the test suite.
[x] Changes were documented in the changelog (pending section).
[x] If necessary, changes were made to the documentation (eg new pipeline).

codecov[bot] commented 9 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (core-refacto@b3274b5). Click here to learn what that means.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## core-refacto #224 +/- ## =============================================== Coverage ? 96.58% =============================================== Files ? 254 Lines ? 8356 Branches ? 0 =============================================== Hits ? 8071 Misses ? 285 Partials ? 0 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.