Topic labeling

[carschno]

Currently, the score of each word per cluster is computed like this:

score_{cluster}({word}) = \sum_{cluster-doc} (TF-IDF(word) * distance(centroid, doc))

Where

• IDF is based on the entire corpus
• distance is the cosine distance between the centroid and the document embedding.

Alternative options to investigate:

• TF-IDF based on documents in cluster, instead of entire corpus
• different methods for computing distance
• assign different weights to TF-IDF and to distance
• LLM-generated topic descriptions, as used by Bertopic; also see this Colab notebook
• [...]

Design idea:

• extract a KeywordExtractor class that holds a TfIdfVectorizer and Corpus object
  • fit(): fit vectorizer to documents in corpus
  • top_words(corpus) extract top words for a corpus. The vectorizer is fitted on KeywordExtractor.corpus, whereas the corpus can be any (sub-)corpus.
• remove Corpus._vectorizer attribute
  • introduce Corpus.top_words attribute to store top words
  • re-consider how Corpus.label attribute is used and set
  • introduce Corpus.collection and/or more details about how corpus was retrieved

[carschno]

• vectorizer can be fitted on various (sub-)corpora: mark which cluster is has been fitted on; how to keep track for corpora inheriting a vectorizer?
• check if the weighted distance makes sense; does the distance out-weigh the TF-IDF?
• why are the distances between passages and centroid different from the ones Weaviate computes?