Consider to add named entity recognition

apicrafter / metacrafter

Metadata and data identification tool and Python library. Identifies PII, common identifiers, language specific identifiers. Fully customizable and flexible rules

Apache License 2.0

44 stars 5 forks source link

Consider to add named entity recognition #1

Open ivbeg opened 2 years ago

ivbeg commented 2 years ago

Named entity recognitions technology helps to identify named objects inside texts.

Strong

allows to identify objects inside text blobs
could allow to support more named entities (identifiers)

Weakness

could be very slow
need to prepare PII and identifier rules for recognition

Possible implementation - Slovnet https://github.com/natasha/slovnet

ivbeg commented 2 years ago

Presidio looks like possible NER engine. The ways to implement:

support analysis of list of fields
support analysis of any string fields with length greater than max_len parameter. Support this parameter.
link NER entities to semantic types registry