adds preprocessing functions for normalizing bullet points in lists, removing html tags, and removing bracketed contents like in-line citations
adds a way to create preprocessing pipelines composed of multiple functions applied sequentially to input text
renames functions and restructures imports for improved flexibility and clarity of use; in most cases, this will entail replacing an underscore with a period, e.g. preprocessing.normalize_whitespace() => preprocessing.normalize.whitespace()
some funcs' args were likewise renamed and standardized; for example, all remove functions had their (optional) second argument renamed from replace_with => repl, and remove.punctuation(text, marks=".?!") => remove.punctuation(text, only=[".", "?", "!"])
improves and parametrizes preprocessing tests
Motivation and Context
I'm looking to live up to textacy's tagline of "NLP, before and after spaCy", where preprocessing is the bulk of the "before". These changes improve and extend those capabilities.
How Has This Been Tested?
tests added, and all tests pass
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation, and I have updated it accordingly.
Description
preprocessing.normalize_whitespace()=>preprocessing.normalize.whitespace()removefunctions had their (optional) second argument renamed fromreplace_with=>repl, andremove.punctuation(text, marks=".?!")=>remove.punctuation(text, only=[".", "?", "!"])Motivation and Context
I'm looking to live up to textacy's tagline of "NLP, before and after spaCy", where preprocessing is the bulk of the "before". These changes improve and extend those capabilities.
How Has This Been Tested?
tests added, and all tests pass
Screenshots (if appropriate):
Types of changes
Checklist: