adds new and extends existing information extraction functionality
adds extract.terms(): flexibly combine ngrams, entities, and/or noun chunks terms into a single collection, with optional deduplication of identical spans
extends extract.regex_matches(): allow for multiple ways for matches to align to spacy tokenization ("strict", "contract", or "expand")
extends extract.ngrams(): allow for specifying one or multiple n values, whose corresponding collections of ngrams are concatenated
moves and refactors keyterm utility functions up a level to extract.utils, where they may be used to flexibly transform a sequence of spans into a sequence of strings, plus the cleaning/aggregation of those results
adds new, moves+extends existing functionality wrt representing documents in various forms into representations subpackage
consolidates and totally reworks networks functionality in representations.network module
adds build_cooccurrence_network() function that transforms ordered sequence of strings (or a sequence of such sequences) into a graph with nodes for each unique string and edges to other strings that co-occurred
adds build_similarity_network() function that transforms a sequence of strings (or a sequence of such sequences) into a graph with nodes as top-level sequence elements and edges to all other elements weighted by their pairwise similarity measure
removes obsolete network.py pkg-level module and duplicative extract.keyterms.graph_base.py module
refines vectorizer configuration, and moves from vsm.vectorizers to representations.vectorizers module
for both Vectorizer and GroupVectorizer, applying global inverse document frequency weights is now handled by a single arg: idf_type: Optional[str], rather than a combination of apply_idf: bool, idf_type: str; in the same way, applying document-length weight normalizations is handled by dl_type: Optional[str] instead of apply_dl: bool, dl_type: str
adds type hints to all user-facing interfaces
adds representations.sparse_vec module for higher-level access to documents vectorization
adds build_doc_term_matrix() and build_grp_term_matrix() functions, for when you just need to transform a collection of documents and don't want to bother with object-oriented vectorizers
moves, streamlines, and extends spaCy Doc extensions from spacier to a pkg-level module
refactors and improves performance of Doc._.to_bag_of_words() and Doc._.to_bag_of_terms() methods, leveraging and mirroring related functionality in extract.words() and extract.terms()
removes some redundant / awkward extensions:
._.n_tokens => len(Doc)
._.to_tagged_text() => NA ... this was a very old holdover originally inspired by some NLTK workflows that don't get used anymore
._.to_terms_list() => extract.terms(doc) or, if you've done import textacy.extract, that function is also available as a Doc extension via ._.terms()
._.to_semantic_network() => use an appropriate implementation in representations.networks
changes args on Corpus.word_counts() and Corpus.word_doc_counts() to match usage on Doc._.to_bag_of_words()
improves lots of documentation and tests
Motivation and Context
Despite best efforts, certain corners of textacy have gathered some nasty cobwebs. Functionality for representing documents as networks was particularly gross, suffering from code duplication in two different modules and awkward/unclear intentions behind some functions. The vsm subpackage was also long-neglected — it didn't even have type annotations! — and suffered from clunky, error-prone weighting configuration. Lastly, the spacier Doc extensions were in a weird, frozen place, implemented while spaCy was still expanding upon its extension / customization functionality and while I was still getting used to it.
How Has This Been Tested?
many tests added and improved, all pass
Screenshots (if appropriate):
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation, and I have updated it accordingly.
Description
extract.terms(): flexibly combine ngrams, entities, and/or noun chunks terms into a single collection, with optional deduplication of identical spansextract.regex_matches(): allow for multiple ways for matches to align to spacy tokenization ("strict", "contract", or "expand")extract.ngrams(): allow for specifying one or multiplenvalues, whose corresponding collections of ngrams are concatenatedextract.utils, where they may be used to flexibly transform a sequence of spans into a sequence of strings, plus the cleaning/aggregation of those resultsrepresentationssubpackagerepresentations.networkmodulebuild_cooccurrence_network()function that transforms ordered sequence of strings (or a sequence of such sequences) into a graph with nodes for each unique string and edges to other strings that co-occurredbuild_similarity_network()function that transforms a sequence of strings (or a sequence of such sequences) into a graph with nodes as top-level sequence elements and edges to all other elements weighted by their pairwise similarity measurenetwork.pypkg-level module and duplicativeextract.keyterms.graph_base.pymodulevsm.vectorizerstorepresentations.vectorizersmoduleVectorizerandGroupVectorizer, applying global inverse document frequency weights is now handled by a single arg:idf_type: Optional[str], rather than a combination ofapply_idf: bool, idf_type: str; in the same way, applying document-length weight normalizations is handled bydl_type: Optional[str]instead ofapply_dl: bool, dl_type: strrepresentations.sparse_vecmodule for higher-level access to documents vectorizationbuild_doc_term_matrix()andbuild_grp_term_matrix()functions, for when you just need to transform a collection of documents and don't want to bother with object-oriented vectorizersDocextensions fromspacierto a pkg-level moduleDoc._.to_bag_of_words()andDoc._.to_bag_of_terms()methods, leveraging and mirroring related functionality inextract.words()andextract.terms()._.n_tokens=>len(Doc)._.to_tagged_text()=> NA ... this was a very old holdover originally inspired by some NLTK workflows that don't get used anymore._.to_terms_list()=>extract.terms(doc)or, if you've doneimport textacy.extract, that function is also available as aDocextension via._.terms()._.to_semantic_network()=> use an appropriate implementation inrepresentations.networksCorpus.word_counts()andCorpus.word_doc_counts()to match usage onDoc._.to_bag_of_words()Motivation and Context
Despite best efforts, certain corners of
textacyhave gathered some nasty cobwebs. Functionality for representing documents as networks was particularly gross, suffering from code duplication in two different modules and awkward/unclear intentions behind some functions. Thevsmsubpackage was also long-neglected — it didn't even have type annotations! — and suffered from clunky, error-prone weighting configuration. Lastly, the spacier Doc extensions were in a weird, frozen place, implemented while spaCy was still expanding upon its extension / customization functionality and while I was still getting used to it.How Has This Been Tested?
many tests added and improved, all pass
Screenshots (if appropriate):
Types of changes
Checklist: