[Bug]: `openai==1.47` breaks CI

[jamesbraza]

What happened?

Something about openai==1.47.0 and 1.47.1 breaks paper-qa's CI. Our CI (run) starts blowing up with the below error.

Relevant log output

litellm.exceptions.APIError: litellm.APIError: APIError: OpenAIException - Connection error.

Twitter / LinkedIn details

No response

[CGH20171006]

This problem was not solved in my case by changing the version of openai： windows10+python3.11 openai-1.45.0

settings = Settings(
    llm="gpt-4o-mini",
    summary_llm="gpt-4o-mini",
    paper_directory="D:\\Programing\\paper-qa",
    verbosity=3
)

query = "What manufacturing challenges are unique to bispecific antibodies?"

try:
    response = ask(query, settings)
    print(response)
except Exception as e:
    print(f"An error occurred: {e}")

error message

PaperQA version: 5.0.7
[10:29:07] Beginning agent 'fake' run with question 'What manufacturing challenges are unique to bispecific antibodies?' and full query
           {'query': 'What manufacturing challenges are unique to bispecific antibodies?', 'id':
           UUID('562dea78-4538-496d-a77b-71a10725892a'), 'settings_template': None, 'settings': {'llm': 'gpt-4o-mini', 'llm_config': None,
           'summary_llm': 'gpt-4o-mini', 'summary_llm_config': None, 'embedding': 'text-embedding-3-small', 'embedding_config': None,
           'temperature': 0.0, 'batch_size': 1, 'texts_index_mmr_lambda': 1.0, 'index_absolute_directory': False, 'index_directory':
           WindowsPath('C:/Users/20171006/.pqa/indexes'), 'index_recursively': True, 'verbosity': 3, 'manifest_file': None,
           'paper_directory': 'D:\\Programing\\paper-qa', 'answer': {'evidence_k': 10, 'evidence_detailed_citations': True,
           'evidence_retrieval': True, 'evidence_summary_length': 'about 100 words', 'evidence_skip_summary': False, 'answer_max_sources':
           5, 'answer_length': 'about 200 words, but can be longer', 'max_concurrent_requests': 4, 'answer_filter_extra_background': False},
           'parsing': {'chunk_size': 3000, 'use_doc_details': True, 'overlap': 100, 'citation_prompt': 'Provide the citation for the
           following text in MLA Format. Do not write an introductory sentence. If reporting date accessed, the current year is
           2024\n\n{text}\n\nCitation:', 'structured_citation_prompt': "Extract the title, authors, and doi as a JSON from this MLA
           citation. If any field can not be found, return it as null. Use title, authors, and doi as keys, author's value should be a list
           of authors. {citation}\n\nCitation JSON:", 'disable_doc_valid_check': False, 'chunking_algorithm':
           <ChunkingOptions.SIMPLE_OVERLAP: 'simple_overlap'>}, 'prompts': {'summary': 'Summarize the excerpt below to help answer a
           question.\n\nExcerpt from {citation}\n\n----\n\n{text}\n\n----\n\nQuestion: {question}\n\nDo not directly answer the question,
           instead summarize to give evidence to help answer the question. Stay detailed; report specific numbers, equations, or direct
           quotes (marked with quotation marks). Reply "Not applicable" if the excerpt is irrelevant. At the end of your response, provide
           an integer score from 1-10 on a newline indicating relevance to question. Do not explain your score.\n\nRelevant Information
           Summary ({summary_length}):', 'qa': 'Answer the question below with the context.\n\nContext (with relevance
           scores):\n\n{context}\n\n----\n\nQuestion: {question}\n\nWrite an answer based on the context. If the context provides
           insufficient information reply "I cannot answer."For each part of your answer, indicate which sources most support it via
           citation keys at the end of sentences, like {example_citation}. Only cite from the context below and only use the valid keys.
           Write in the style of a Wikipedia article, with concise sentences and coherent paragraphs. The context comes from a variety of
           sources and is only a summary, so there may inaccuracies or ambiguities. If quotes are present and relevant, use them in the
           answer. This answer will go directly onto Wikipedia, so do not add any extraneous information.\n\nAnswer ({answer_length}):',
           'select': 'Select papers that may help answer the question below. Papers are listed as $KEY: $PAPER_INFO. Return a list of keys,
           separated by commas. Return "None", if no papers are applicable. Choose papers that are relevant, from reputable sources, and
           timely (if the question requires timely information).\n\nQuestion: {question}\n\nPapers: {papers}\n\nSelected keys:', 'pre':
           None, 'post': None, 'system': 'Answer in a direct and concise tone. Your audience is an expert, so be highly specific. If there
           are ambiguous terms or acronyms, first define them.', 'use_json': False, 'summary_json': 'Excerpt from
           {citation}\n\n----\n\n{text}\n\n----\n\nQuestion: {question}\n\n', 'summary_json_system': 'Provide a summary of the relevant
           information that could help answer the question based on the excerpt. Respond with the following JSON format:\n\n{{\n  "summary":
           "...",\n  "relevance_score": "..."\n}}\n\nwhere `summary` is relevant information from text - {summary_length} words and
           `relevance_score` is the relevance of `summary` to answer question (out of 10).\n'}, 'agent': {'agent_llm': 'gpt-4o-2024-08-06',
           'agent_llm_config': None, 'agent_type': 'fake', 'agent_config': None, 'agent_system_prompt': 'You are a helpful AI assistant.',
           'agent_prompt': 'Use the tools to answer the question: {question}\n\nThe {gen_answer_tool_name} tool output is visible to the
           user, so you do not need to restate the answer and can simply terminate if the answer looks sufficient. The current status of
           evidence/papers/cost is {status}', 'return_paper_metadata': False, 'search_count': 8, 'wipe_context_on_answer_failure': True,
           'timeout': 500.0, 'should_pre_search': False, 'tool_names': None, 'index_concurrency': 30}, 'md5':
           'd9b11506128c475035509ae3cfc1addb'}, 'docs_name': None}.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

An error occurred: litellm.APIError: APIError: OpenAIException - Connection error.
Received Model Group=gpt-4o-mini
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3
(paperqa) PS D:\Programing\paper-qa>

[ishaan-jaff]

@jamesbraza @CGH20171006 is this just on CI/CD ? Can you give us more steps to repro this

[jamesbraza]

I found pinning openai<1.47 fixed the issue for me reliably in https://github.com/Future-House/paper-qa/pull/466. You can try playing around with our unit tests there @ishaan-jaff, and thanks for investigating this 👍

Note you will want to delete our test cassettes (at tests/cassettes) because they cache responses and mask the issue.

[CGH20171006]

@jamesbraza @CGH20171006 is this just on CI/CD ? Can you give us more steps to repro this

I use python to run this，i can give your all the code expect the api key:

import os
from paperqa import Docs,ask
from paperqa.settings import Settings, AgentSettings, AnswerSettings
os.environ["OPENAI_API_KEY"] = "sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
os.environ["OPENAI_API_BASE"] = "https://api.fast-tunnel.one/v1"

settings = Settings(
    llm="gpt-4o-mini",
    summary_llm="gpt-4o-mini",
    paper_directory="D:\\Programing\\paper-qa",
    verbosity=3
)

query = "What manufacturing challenges are unique to bispecific antibodies?"

try:
    response = ask(query, settings)
    print(response)
except Exception as e:
    print(f"An error occurred: {e}")

[JamesHutchison]

Was seeing this issue and pinning to 1.45.0 fixed it.

Whatever the cause, it happens almost immediately after a request is made, ignoring retries

I don't think this is CI related, was seeing this in a Codespace. The issue is either random / intermittent or happens on a cadence such as every other request. When I was debugging on an integration test using the pytest hot reloading daemon, I'd see this error, then see a different error further along, then see this error, then see a different error...

There is no status code

[ishaan-jaff]

I don't think this is CI related, was seeing this in a Codespace.

@JamesHutchison can you help me repro

• what environment were you on ?
• could you share code to repro ?

[ishaan-jaff]

I see nothing about this on OpenAI Python Releases https://github.com/openai/openai-python/releases or issues: https://github.com/openai/openai-python/issues

[jamesbraza]

For me in paper-qa, we don't depend on openai, just litellm, so it's some internal interaction between litellm and the openai package. This error started appearing like a switch when upgrading our indirect dependency from openai==1.46.1 to openai==1.47.0.

Also, I observe this error type seems to only happen when multiprocessing via pytest-xdist. https://github.com/BerriAI/litellm/issues/4032 is another open issue about basically LiteLLM + multiprocessing pytest-xdist + Anthropic failures.

I think the core of this issue is that something about LiteLLM's design is vulnerable to multiprocessing failures, and it seems something about the latest openai package's minor version exposes it in a new way.