glass clustering flow + util scripts

[Juan-Mateos]

Closes #19

---

Checklist:

• [x] I have refactored my code out from notebooks/
• [x] I have run flake8 and addressed any linter erors
• [x] I have checked the code runs
• [ ] I have tested the code
• [x] I have run pre-commit and addressed any issues not automatically fixed
• [x] I have rebased onto dev (or merged any new changes from dev)
• [x] I have documented the code
  • [x] Major functions have docstrings
  • [x] Appropriate information has been added to READMEs
• [x] I have explained the feature in this PR or (better) in output/reports/
• [x] I have requested a code review

[Juan-Mateos]

What this does

We cluster company descriptions in SIC codes. Flow and utilities scripts can be found in industrial_taxonomy/pipeline/cluster_glass.

The user can choose the minimum sector size to include in the clustering (defaults to 1000)

The flow includes branching for different values of an assigned_shares parameter that controls the number of companies within a SIC to assign into clusters. We can test performance for different parameter values downstream.

Before running

• Install graph_tool
• Install topsbm by running gh repo clone martingerlach/hSBM_Topicmodel inside industrial_taxonomy/pipeline/glass_clusters (I couldn't import the topic model when installing via pip @bishax).
  • You will also need to fix a bug in the model.clusters method in the sbmtm.py script (see here for more information).

Running

• The full run including a default grid search that extracts the top [10,100,500,0.5,"all"] business descriptions for cluster took ca. 3.5 hours in my machine. If you just want to check that the code runs for you, run the flow with...
  • --test-mode=True so it only runs on three SIC codes,
  • --assigned_shares='[10,1000'] so that only does two extractions per SIC code.
  • This should take 5 minutes or so
  • And get the full extracted data from the latest run of the flow.

[Juan-Mateos]

I did the following:

Generic stuff

• Removed the instructions to clone and fix hsbmtm from the README
• added the sbmtm.py repo

flow.py

• Added more information about the purpose of the flow
• Fixed typos in docstrings
• Removed empty and unnecessary steps (leftover from a previous version of the flow)
• I separated topic modelling and description clusterings into two steps because it is unnecessary to train a new topic model for each assignment strategy (we only use the strategy to decide how many documents to cluster inside a topic model)
• test-mode only runs when not with --production.
• Kept all_sectors_corpora as a dict (converting it to a list was a leftover from an older version of the flow where I branched the flow over sectors)
• This simplified subsequent steps as you suggested
• I clarified the meaning of the assigned-shares parameter
• I created {assigned_shares:models, clusters} dicts in the cluster_glass_descriptions step as suggested and used the join step to combine them into a single dict.

topic_utils.py

• Subsumed fit_model code within fit_model_sector
• Fixed type typos

utils.py

• Removed unnecessary partial in the tokenising pipe
• Changed gl_sic4 name
• Moved glass_sic4_lookup to the glass_house getters and added new info to the docstring
  • Changed variable names and apply to slice
  • I also fixed the typing bug in get_sector
• Changed variable names in the dict comprehensions

requirements.txt

• I removed the pip install topsbm thing.

I didn't do the following

• I tried using the conda decorator to create a virtual environment with graph-tool but this raised a pip install error so I decided to follow your advice in Kuebiko: "stick to using your project’s main environment and only use Metaflow’s conda decorators when you definitely need them."
  • I tried adding AB's refactored version of sbmtm.py but that raised an error I assume is linked to subsequent changes in graph-tool:
  • Error message: minimize_nested_blockmodel_dl() got an unexpected keyword argument 'deg_corr'
  • I didn't decompose lookup creation and filtering into two functions because lookup creation would have amounted to a thin wrapper doing {k:v for k,v in dict.items()}

Misc

• The reason to strip_nes is because we wouldn't want ngrams including generic NE names to appear in sector names generated later. Admittedly, this is unlikely to happen + we could also strip them when we name the sectors. I don't feel strongly either way.

Points for clarification

• What do you mean by "input data should be a parameter" (to pipe)?

[Juan-Mateos]

What I have done:

flow.py

• Removed hSBM import
• Relative paths for util and topic_util imports
• Removed redundant self.sectors
• Fixed various typing problems

utils.py

• tokenised filtering pipeline takes data instead of a getter
• added big_ to sector_tokens_lookup

misc

• Renamed utils/utils.py as collections.py

What I haven't done

• Branching for each sector

[Juan-Mateos]

What I have done:

flow.py

• Tested with mypy and fixed all typing errors
• Updated the docstrings
• Added glass_sic4_lookup as a parameter into make_sector_corpora
• Removed duplicated self.sectors assignment
• Removed comma from topic_utils import

utils.py

• Turned glass_sic4_lookup into a parameter for make_sector_corpora

Given issues loading saved models I opted to leave the sbmtm imports as they were previously.

I ran the flow on test-mode and it works.

[Juan-Mateos]

I turned get_descriptions_tokenised into an argument into make_sector_corpora

I ran it in test-mode and it works.