openai.ContentFilterFinishReasonError

[matheus-rossi]

Describe the bug When scraping pages, randomly get this error:

Could not parse response content as the request was rejected by the content filter

response = self.root_client.beta.chat.completions.parse(**payload)
│ │ │ │ │ │ └ {'messages': [{'content': '\nYou are a website scraper and you have just scraped the\nfollowing content from a website conver...
│ │ │ │ │ └ <function Completions.parse at 0x7f075673b640>
│ │ │ │ └ <openai.resources.beta.chat.completions.Completions object at 0x7f0731467310>
│ │ │ └ <openai.resources.beta.chat.chat.Chat object at 0x7f0731466fe0>
│ │ └ <openai.resources.beta.beta.Beta object at 0x7f0731466d40>
│ └ <openai.OpenAI object at 0x7f073110f010>
└ ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x7f07314658a0>, async_client=<openai.resources.ch...
File "/root/.cache/pypoetry/virtualenvs/platform-backend-data-tq7C0_9c-py3.10/lib/python3.10/site-packages/openai/resources/beta/chat/completions.py", line 140, in parse
return _parse_chat_completion(
└ <function parse_chat_completion at 0x7f0756700d30>
File "/root/.cache/pypoetry/virtualenvs/platform-backend-data-tq7C0_9c-py3.10/lib/python3.10/site-packages/openai/lib/_parsing/_completions.py", line 75, in parse_chat_completion
raise ContentFilterFinishReasonError()
└ <class 'openai.ContentFilterFinishReasonError'>

openai.ContentFilterFinishReasonError: Could not parse response content as the request was rejected by the content filter

To Reproduce Try to scrape this page:

https://news.bms.com/news/details/2016/First-Presentation-of-Two-Year-Overall-Survival-Data-for-Opdivo-nivolumab-in-Combination-with-Yervoy-ipilimumab-Showed-Superior-Efficacy-Versus-Yervoy-Alone-in-Advanced-Melanoma/default.aspx

Expected behavior Extracted Content

Additional context Add any other context about the problem here.

[VinciGit00]

can you share the code please?