get_topics function

[skewballfox]

• currently there are close to ~12,000 annotations spread across ~900 documents, divided into 239 cases
  • this might be too many classifications for a model to handle, if so we may cluster the cases into topics
• topics are something tosdr does to group cases together under certain ideas (such as dealing with the concept of "ownership"
  • there are only 28 topics, excluding "deprecated", and not all of those may be necessary for the 239 cases we have
  • we can then take the scoring field for the case (which is currently useless for our situation), to then expand the topics out to 3 categories: good,bad,or neutral
  • this would reduces 239 cases to at most 84 hierarchical classifications
• topics aren't available via the api so here's what we do:
  1. we create an .env file with credentials for the edit.tosdhr site, where the list of topics is available
  2. we use request.session to create a session, and use python-dotenv to load the credentials for the site
  3. we make a request of the page which list all the topics
  4. we use beautiful soup to parse the content of the page,getting a list of urls and names for the list of topics
     • if we have a way of identifying topics which aren't relevant(such as deprecated) we use them here
     • this might be where we want to use some kind of cache loop the way we do for services
  5. for each topic, we use the existing session to visit the url of associated with topic
  6. we use beautiful soup to parse the body of the response to get the list of cases
  7. we check the cases for that topic against the set of cases we have annotations for.
     • if at least one case under that topic is in the set, we store it in a dict where the key is the numeric topic_id (part of the url), and the value is the the list or set of cases
     • (set of cases may be better, and python supports set operations like union and intersection)
  8. we store this dict to swap out with the cases at tokenization time.

[skewballfox]

the .env file is already present, the code we have is pretty much done down to 4(which is halfway done) if I need to add any information I can post it on discord or here, your preference. I recommend here, especially if the chat is active to make it easier to reference all the relevant information. ping me on discord if you need anything, that's the only way I get a notification