10 cooccurrence taxonomy

[emily-bicks]

---

Description

Builds pipeline to generate taxonomy using term cooccurrences

Fixes #11

Instructions for Reviewer

To test: run scripts with flag --test to run on small sample of dataset. NOTE: if test_mode is not used, running the script will take a long time and overwrite the outputs on AWS.

Please pay special attention to: pipeline/

• pipeline/taxonomy_development/build_cooccurrence_network.py: generates and saves the term co-occurrence network NOTE: you will need to have checked out 78_edge_attributes on nesta_ds_utils as the edge attribute feature of the cooccurrence network hasn't yet been merged into dev
• pipeline/community_detection.py: builds and outputs taxonomy based on network
• disregard pipeline/community_detection_flow.py for now: this is the same as the previous script but refactored into a flow. Doesn't currently work due to the issue with the batch environment, but is only needed for iterating on the taxonomy parameters.

SOME THINGS TO NOTE WITH TAXONOMY THAT MAY REQUIRE ITERATION:

• Starting resolution and resolution increments have a significant effect on the taxonomy assignments
• Most entities don't get assigned to more than 3/4 splits - this is due to the min size of a grouping being set at 10, and is significantly influenced by the starting resolution and resolution increments

There are also new getters:

• getters/cooccurrence_network.py: retrieves networkx graph
• getters/taxonomies.py: retrieves config and taxonomy dataframe

Checklist:

• [X] I have refactored my code out from notebooks/
• [X] I have checked the code runs
• [X] I have tested the code
• [X] I have run pre-commit and addressed any issues not automatically fixed
• [X] I have merged any new changes from dev
• [ ] I have documented the code
  • [X] Major functions have docstrings
  • [ ] Appropriate information has been added to READMEs
• [X] I have explained this PR above
• [x] I have requested a code review