Open jamesleverage opened 9 months ago
Thanks for sharing! It seems that with openai 1.0.0 there were breaking changes to the API which need to be updated in BERTopic. I'll make sure to fix it in this PR since there were more OpenAI updates there.
@jamesleverage If I'm not mistaken, you can use openai 0.28 instead of 0.38 and I believe it should be working. However, I just pushed a fix to the PR mentioned above that should make it work with openai >= 1.0. In the upcoming release of BERTopic, openai < 1.0 will not be supported anymore.
I used openai==0.28 and getting this error at this line after running for 10+ minutes:
representation_model = OpenAI(model="gpt-3.5-turbo", chat=True)
model = BERTopic(representation_model=representation_model) topics, probs = model.fit_transform(smaller_docs_list). <==== This is where the execution hangs.
Error message:
=========================================================================== TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
ReadTimeoutError Traceback (most recent call last) ReadTimeoutError: HTTPSConnectionPool(host='api.openai.com', port=443): Read timed out. (read timeout=600)
During handling of the above exception, another exception occurred:
=========================================================================== The old release is causing a problem.
Is there something I can add to my code to make this work? Or should I just wait for the latest PR fix?
For simple prompting to OpenAI, this definition worked with the latest openai: `` def get_completion( prompt, model="gpt-3.5-turbo", temperature=0):
messages = [{"role": "user", "content":prompt}]
client = OpenAI(api_key=OPENAI_API_KEY)
response = client.chat.completions.create( model=model, messages=messages, temperature=0 )
response_message = response.choices[0].message.content
return response_message ``
Is there something I can add to my code to make this work? Or should I just wait for the latest PR fix?
If 0.28 is currently not working, then I would wait until the PR fix. You can already download it if you want like this:
pip install git+https://github.com/MaartenGr/BERTopic.git@refs/pull/1572/head
Adding to the discussion, while we wait for the PR fix, I'm trying to do the labeling after training BERTopic models. Would it be possible to adjust the number of representative documents when we use get_representative_docs
?
@linxule .get_representative_docs
is a function that does no calculations with respect to extracting the most representative documents. For that, you would have to use the internal ._extract_representative_docs
which is used to calculate which documents are most representative of a given topic. Do note though that since this is a private function, breaking changes might appear in future releases and not additional official support can be given.
I believe you can use it as follows:
import pandas as pd
documents = pd.DataFrame(
{
"Document": docs,
"ID": range(len(docs)),
"Topic": None,
"Image": None
}
)
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topic_model.topics_,
nr_repr_docs=10
)
Where docs
are your input documents. I have not tested this so there might be a few mistakes there but the general principle should be solid.
Hi @MaartenGr ,
I tried the solution you suggested but encountered some issues
I ran
import pandas as pd
# Assuming df_dict and models are defined in an accessible scope
# df_dict: Dictionary of DataFrames
# models: Dictionary of BERTopic models
def extract_representative_documents(df_name, nr_repr_docs=5):
"""
Extracts representative documents for each topic from a DataFrame specified by df_name.
Parameters:
- df_name: The name of the DataFrame within df_dict.
- nr_repr_docs: Number of representative documents to extract for each topic (default is 10).
Returns:
- A DataFrame with the representative documents and their associated topics.
"""
if df_name not in df_dict:
raise ValueError(f"DataFrame with name '{df_name}' not found in df_dict")
if df_name not in models:
raise ValueError(f"BERTopic model with name '{df_name}' not found in models")
# Access the documents from the specified DataFrame
docs = df_dict[df_name]['Post_Content']
# Create a DataFrame for the documents
documents = pd.DataFrame(
{
"Document": docs,
"ID": range(len(docs)),
"Topic": None,
"Image": None
}
)
# Extract representative documents using the BERTopic model
repr_docs, _, _, _ = models[df_name]._extract_representative_docs(
models[df_name].c_tf_idf_,
documents,
models[df_name].topics_,
nr_repr_docs=nr_repr_docs
)
return repr_docs
# Example usage
representative_docs = extract_representative_documents(df_name)
I got
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
[<ipython-input-40-80fe2672c770>](https://localhost:8080/#) in <cell line: 1>()
----> 1 representative_docs = extract_representative_documents(df_name)
2 representative_docs
[<ipython-input-37-023df3f7fa07>](https://localhost:8080/#) in extract_representative_documents(df_name, nr_repr_docs)
35
36 # Extract representative documents using the BERTopic model
---> 37 repr_docs, _, _, _ = models[df_name]._extract_representative_docs(
38 models[df_name].c_tf_idf_,
39 documents,
[/usr/local/lib/python3.10/dist-packages/bertopic/_bertopic.py](https://localhost:8080/#) in _extract_representative_docs(self, c_tf_idf, documents, topics, nr_samples, nr_repr_docs, diversity)
3689 # Sample documents per topic
3690 documents_per_topic = (
-> 3691 documents.drop("Image", axis=1, errors="ignore")
3692 .groupby('Topic')
3693 .sample(n=nr_samples, replace=True, random_state=42)
[/usr/local/lib/python3.10/dist-packages/pandas/core/groupby/groupby.py](https://localhost:8080/#) in sample(self, n, frac, replace, weights, random_state)
4334 sampled_indices.append(grp_indices[grp_sample])
4335
-> 4336 sampled_indices = np.concatenate(sampled_indices)
4337 return self._selected_obj.take(sampled_indices, axis=self.axis)
4338
/usr/local/lib/python3.10/dist-packages/numpy/core/overrides.py in concatenate(*args, **kwargs)
ValueError: need at least one array to concatenate
Besides the proposed solution, is there any way to use topic_model.get_representative_docs()
and specify the number of representative documents? This approach seems to be default to 3 representative documents?
@linxule
Besides the proposed solution, is there any way to use topic_model.get_representative_docs() and specify the number of representative documents? This approach seems to be default to 3 representative documents?
As mentioned above, topic_model.get_representative_docs()
does not actually calculate which documents are most representative as that is done during topic_model.fit(docs)
. Instead, topic_model.get_representative_docs()
simply gets the previously calculated representative documents in a nice format. As a result, it is simply not possible to get more than 3 representative documents that way since the trained documents are not saved within the topic model. The reason for this is that saving training data within a model is something that we should generally prevent, especially if the data is large.
Instead, you can fix the error you ran into with the following code. I just tested it and it should work to extract, for example, the top 10 topics:
import pandas as pd
documents = pd.DataFrame(
{
"Document": docs,
"ID": range(len(docs)),
"Topic": topic_model.topics_,
"Image": None
}
)
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topic_model.topic_labels_,
nr_repr_docs=10
)
Note that what is happening in the above code is that the documents are passed to the function which is not the case with topic_model.get_representative_docs
.
@MaartenGr Thank you for your quick response.
I ran
# Access the the model and documents
docs = df_dict[df_name]['Post_Content']
topic_model = models[df_name]
# Create a DataFrame for the documents
documents = pd.DataFrame(
{
"Document": docs,
"ID": range(len(docs)),
"Topic": topic_model.topics_,
"Image": None
}
)
# Extract representative documents using the BERTopic model
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topic_model.topics_,
nr_repr_docs=10
)
Ang got
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-98-cefb73aba3fa> in <cell line: 16>()
14
15 # Extract representative documents using the BERTopic model
---> 16 repr_docs, _, _, _ = topic_model._extract_representative_docs(
17 topic_model.c_tf_idf_,
18 documents,
/usr/local/lib/python3.10/dist-packages/bertopic/_bertopic.py in _extract_representative_docs(self, c_tf_idf, documents, topics, nr_samples, nr_repr_docs, diversity)
3700 repr_docs_mappings = {}
3701 repr_docs_ids = []
-> 3702 labels = sorted(list(topics.keys()))
3703 for index, topic in enumerate(labels):
3704
AttributeError: 'list' object has no attribute 'keys'
So I adjusted it to
# Access the model and documents
docs = df_dict[df_name]['Post_Content']
topic_model = models[df_name]
# Retrieve the topics as a dictionary (replace get_topics() with the correct method)
topics_dict = topic_model.get_topics() # This should be a dictionary
# Create a DataFrame for the documents
documents = pd.DataFrame(
{
"Document": docs,
"ID": range(len(docs)),
"Topic": topic_model.topics_,
"Image": None
}
)
# Extract representative documents using the BERTopic model
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topics_dict, # Use the topics dictionary
nr_repr_docs=10
)
This worked. Do you have any comments on this approach? Am I missing anything?
Thank you again for your help!
The reason for your error is that you did not copy my example as showed. In ._extract_representative_docs
you should use topic_model.topic_labels_
instead of using topic_model.topics_
.
Don't do this:
# Extract representative documents using the BERTopic model
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topic_model.topics_,
nr_repr_docs=10
)
Do this:
repr_docs, _, _, _ = topic_model._extract_representative_docs(
topic_model.c_tf_idf_,
documents,
topic_model.topic_labels_,
nr_repr_docs=10
)
Thank you so much for spotting the error! It works now!
Is there something I can add to my code to make this work? Or should I just wait for the latest PR fix?
If 0.28 is currently not working, then I would wait until the PR fix. You can already download it if you want like this:
pip install git+https://github.com/MaartenGr/BERTopic.git@refs/pull/1572/head
There is a warning message. !pip install git+https://github.com/MaartenGr/BERTopic.git@refs/pull/1572/head
Collecting git+https://github.com/MaartenGr/BERTopic.git@refs/pull/1572/head Cloning https://github.com/MaartenGr/BERTopic.git (to revision refs/pull/1572/head) to /tmp/pip-req-build-9vcn0uye Running command git clone --filter=blob:none --quiet https://github.com/MaartenGr/BERTopic.git /tmp/pip-req-build-9vcn0uye WARNING: Did not find branch or tag 'refs/pull/1572/head', assuming revision or ref.
@jamesleverage You can try this
!pip install git+https://github.com/MaartenGr/BERTopic.git@6fd3e14fa0867d5d68c580b75a0b40151626e80b
This will install the Commits on Nov 17, 2023 (https://github.com/MaartenGr/BERTopic/pull/1572/commits/6fd3e14fa0867d5d68c580b75a0b40151626e80b)
Hello! working with this collab of BERTopic - Best Practices.ipynb
I am getting this error even working with openai==0.28(It was working before). Sorry i saw the comments but I am confused.
<ipython-input-23-6a1f85215426> in <cell line: 27>()
25
26 # Initialize the OpenAI model for BERTopic
---> 27 openai_model = OpenAI(model="gpt-3.5-turbo", prompt=prompt, chat=True, exponential_backoff=True)
28
29 # All representation models
TypeError: OpenAI.__init__() missing 1 required positional argument: 'client'
@giannisni Thanks for sharing! I just updated the notebook, can you check whether it works?
Hi @MaartenGr it works now thanks. Thought in my copy of notebook, using the same (mine) documents as i did before I am getting this error now, which was not happening. Also can you please explain what is passed exactly on the [DOCUMENTS] in the prompt?
prompt = """ I have a topic that contains the following documents: [DOCUMENTS] The topic is described by the following keywords: [KEYWORDS]
Based on the information above, extract a short but highly descriptive topic label of at most 5 words. Make sure it is in the following format:
topic:
The error:
BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 16385 tokens. However, your messages resulted in 40728 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}
@giannisni Most likely, your documents are simply too big. I would advise applying document truncation using this guide.
@MaartenGr,
I'm experiencing a similar incompatibility issue. I am running bertopic version 0.16.0 and it was running fine until I updated openai. Now, I'm getting import errors no matter which version of openai I tried. I've tried 0.28, 1.10 and the latest version.
Whenever I import bertopic I'm getting the following error:
` Input In [1], in <cell line: 1>() ----> 1 import bertopic
File ~/.conda/envs/py310/lib/python3.9/site-packages/bertopic/init.py:1, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/bertopic/_bertopic.py:49, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/bertopic/representation/init.py:30, in pip install langchain
\n\n"
File ~/.conda/envs/py310/lib/python3.9/site-packages/bertopic/representation/_langchain.py:4, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/init.py:6, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/agents/init.py:2, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/agents/agent.py:16, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/agents/tools.py:8, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/tools/init.py:42, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/tools/vectorstore/tool.py:13, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/chains/init.py:2, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/chains/api/base.py:13, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/chains/api/prompt.py:2, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/prompts/init.py:3, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/prompts/chat.py:10, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/memory/init.py:28, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/memory/vectorstore.py:10, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/vectorstores/init.py:2, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/vectorstores/analyticdb.py:15, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/embeddings/init.py:19, in
File ~/.conda/envs/py310/lib/python3.9/site-packages/langchain/embeddings/openai.py:66, in openai
python package installed, and the
(...)
103
104 """
106 client: Any #: :meta private:
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/main.py:197, in pydantic.main.ModelMetaclass.new()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:506, in pydantic.fields.ModelField.infer()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:436, in pydantic.fields.ModelField.init()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:552, in pydantic.fields.ModelField.prepare()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:663, in pydantic.fields.ModelField._type_analysis()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:808, in pydantic.fields.ModelField._create_sub_type()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:436, in pydantic.fields.ModelField.init()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:552, in pydantic.fields.ModelField.prepare()
File ~/.conda/envs/py310/lib/python3.9/site-packages/pydantic/fields.py:668, in pydantic.fields.ModelField._type_analysis()
File ~/.conda/envs/py310/lib/python3.9/typing.py:852, in _SpecialGenericAlias.subclasscheck(self, cls) 850 return issubclass(cls.origin, self.origin) 851 if not isinstance(cls, _GenericAlias): --> 852 return issubclass(cls, self.origin) 853 return super().subclasscheck(cls)
TypeError: issubclass() arg 1 must be a class `
I've also created a new conda environment and retried it and it still does not work. It only runs when using version 14 or smaller. Do you know what might be going on?
@SebastianSpeer
Based on this line in your error message, you are not using v0.16.0 but v0.15.0:
3 version = "0.15.0"
Please make sure that you are using the newest version of BERTopic.
A related question: can I somehow set the temperature argument when using openAI () to refine topic representations? Now I am using:
representation_model_openai = OpenAI(client, model="gpt-4-turbo-preview", chat=True)
topic_model.update_topics(texts, topics, representation_model=representation_model_openai)
and the topics I got changed slightly every time I ran the code.
A related question: can I somehow set the temperature argument when using openAI () to refine topic representations? Now I am using:
You can use the generator_kwargs
for that (see the docstrings).
and the topics I got changed slightly every time I ran the code.
I would advise checking out the FAQ. It might be that you need to install UMAP from the main branch (I believe a PR updated some things) but I'm not sure, you will have to test.
A related question: can I somehow set the temperature argument when using openAI () to refine topic representations? Now I am using:
You can use the
generator_kwargs
for that (see the docstrings).and the topics I got changed slightly every time I ran the code.
I would advise checking out the FAQ. It might be that you need to install UMAP from the main branch (I believe a PR updated some things) but I'm not sure, you will have to test.
Thanks! I will try out the generator_kwargs
. I have already set a seed for UMAP, and my topics are the same every time, the only difference is the topic representations I get.
prompt = """ I have a topic that contains the following documents: [DOCUMENTS] The topic is described by the following keywords: [KEYWORDS]
Based on the information above, extract a short but highly descriptive topic label of at most 5 words. Make sure it is in the following format: topic:
"""
openai_model = OpenAI(model="gpt-3.5-turbo", exponential_backoff=True, chat=True, prompt=prompt)
All representation models
representation_model = { "OpenAI": openai_model, # Uncomment if you will use OpenAI }
topics, probs = model.fit_transform(smaller_docs_list)
Getting following error message:
You tried to access openai.ChatCompletion, but this is no longer supported in openai>=1.0.0 - see the
When I downgrade opneai to 0.38, this error went away. However, the execution timed out after 600 seconds.