MaartenGr
commented
2 years ago That is currently not possible within BERTopic. Those are intermediate results that are not saved as BERTopic would become a bit too big. Having said that, you can try to expose them as follows:
from bertopic import BERTopic
class BERTopicNew(BERTopic):
def fit_transform(self,
documents: List[str],
embeddings: np.ndarray = None,
y: Union[List[int], np.ndarray] = None) -> Tuple[List[int],
Union[np.ndarray, None]]:
""" Fit the models on a collection of documents, generate topics, and return the docs with topics
Arguments:
documents: A list of documents to fit on
embeddings: Pre-trained document embeddings. These can be used
instead of the sentence-transformer model
y: The target class for (semi)-supervised modeling. Use -1 if no class for a
specific instance is specified.
Returns:
predictions: Topic predictions for each documents
probabilities: The probability of the assigned topic per document.
If `calculate_probabilities` in BERTopic is set to True, then
it calculates the probabilities of all topics across all documents
instead of only the assigned topic. This, however, slows down
computation and may increase memory usage.
Usage:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all')['data']
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs) If you want to use your own embeddings, use it as follows:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
# Create embeddings
docs = fetch_20newsgroups(subset='all')['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=True)
# Create topic model
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs, embeddings)
```
"""
check_documents_type(documents)
check_embeddings_shape(embeddings, documents)
documents = pd.DataFrame({"Document": documents,
"ID": range(len(documents)),
"Topic": None})
# Extract embeddings
if embeddings is None:
self.embedding_model = select_backend(self.embedding_model,
language=self.language)
embeddings = self._extract_embeddings(documents.Document,
method="document",
verbose=self.verbose)
logger.info("Transformed documents to Embeddings")
else:
if self.embedding_model is not None:
self.embedding_model = select_backend(self.embedding_model,
language=self.language)
# Reduce dimensionality with UMAP
if self.seed_topic_list is not None and self.embedding_model is not None:
y, embeddings = self._guided_topic_modeling(embeddings)
umap_embeddings = self._reduce_dimensionality(embeddings, y)
# Cluster UMAP embeddings with HDBSCAN
documents, probabilities = self._cluster_embeddings(umap_embeddings, documents)
# Sort and Map Topic IDs by their frequency
if not self.nr_topics:
documents = self._sort_mappings_by_frequency(documents)
# Extract topics by calculating c-TF-IDF
self._extract_topics(documents)
# Reduce topics
if self.nr_topics:
documents = self._reduce_topics(documents)
self._map_representative_docs(original_topics=True)
probabilities = self._map_probabilities(probabilities, original_topics=True)
predictions = documents.Topic.to_list()
###################################
### CHANGES MADE HERE FOR aph61 ###
###################################
self.sentence_embeddings = embeddings
self.umap_embeddings = umap_embeddings
return predictions, probabilities
In the code above I created a new BERTopic class named `BERTopicNew` that you can use instead of BERTopic. In the most lower part of the code, you will see that I assigned the embeddings that you look for to the `self.` variables as a way to expose them.
In practice, you can run it like this:
```python
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
topic_model = BERTopicNew()
topics, probs = topic_model.fit_transform(docs)and then access the embeddings through:
topic_model.sentence_embeddingsand
topic_model.umap_embeddingsThe above steps are meant for taking the BERTopic package and exposing the sentence embeddings and the embeddings after they have been reduced in dimensionality through UMAP. Does this answer your question?
aph61
You have two versions for BERTopic, the fully integrated with a lot of additional methods, and a step-by-step in the colab "Topic with BERT.ipynb" I use the latter code to make an interactive 3D color image from the standard BERTopic().visualize_topics. You can't run them separately as the topic IDs will not match, and if you have very small topic sizes the topic content won't match. I don't wish to loose the BERTopic methods (very handy) either Is there a way to get the data (embeddings) after the intermediate steps SentenceEmbedding, UMAP, and HDBSCAN in BERTopic()? I couldn't find it as a method; where to start implementing something like dump_umap, dump_word, dump_hdbscan
thanks
Andreas
"Topic with BERT.ipynb" : https://colab.research.google.com/drive/1-SOw0WHZ_ZXfNE36KUe3Z-UpAO3vdhGg?usp=sharing#scrollTo=b8MBLqisBezg)