Builds pipeline to generate taxonomy using term cooccurrences
Fixes #11
Instructions for Reviewer
To test: run scripts with flag --test to run on small sample of dataset. NOTE: if test_mode is not used, running the script will take a long time and overwrite the outputs on AWS.
Please pay special attention to:
pipeline/
pipeline/taxonomy_development/build_cooccurrence_network.py: generates and saves the term co-occurrence network
NOTE: you will need to have checked out 78_edge_attributes on nesta_ds_utils as the edge attribute feature of the
cooccurrence network hasn't yet been merged into dev
pipeline/community_detection.py: builds and outputs taxonomy based on network
disregard pipeline/community_detection_flow.py for now: this is the same as the previous script but refactored into a flow. Doesn't currently work due to the issue with the batch environment, but is only needed for iterating on the taxonomy parameters.
SOME THINGS TO NOTE WITH TAXONOMY THAT MAY REQUIRE ITERATION:
Starting resolution and resolution increments have a significant effect on the taxonomy assignments
Most entities don't get assigned to more than 3/4 splits - this is due to the min size of a grouping being set at 10, and is significantly influenced by the starting resolution and resolution increments
Description
Builds pipeline to generate taxonomy using term cooccurrences
Fixes #11
Instructions for Reviewer
To test: run scripts with flag --test to run on small sample of dataset. NOTE: if test_mode is not used, running the script will take a long time and overwrite the outputs on AWS.
Please pay special attention to: pipeline/
SOME THINGS TO NOTE WITH TAXONOMY THAT MAY REQUIRE ITERATION:
There are also new getters:
Checklist:
notebooks/
pre-commit
and addressed any issues not automatically fixeddev
README
s