Updated ac_dc - Githubissues

bigscience-workshop / data_tooling

Tools for managing datasets for governance and training.

Apache License 2.0

74 stars 48 forks source link

Updated ac_dc #402

Closed ianyu93 closed 2 years ago

ianyu93 commented 2 years ago

This PR includes the following actions:

Move Submodule muliwai from root to ac_dc to allow cleaner local reference, since this is the only directory that is using the submodule.
Uncommented and slightly modified anonymization.py to work with its imports.
Created test_anonymization.py with test cases through faker and num2words.

The following actions are required:

From the test cases, it seems like muliwai.pii_regexes.detect_ner_with_regex_and_context is not working as expected, which requires some check / debugging. Specifically, it seems to only work with global regex, not language specific ones. Please run test_anonymization.py for clarity.
We may need contributors from other languages to provide some test cases in similar format

HugoLaurencon commented 2 years ago

Hey @ianyu93 thank you for this! There are just some small conflicts, can you solve them so I can merge?