Error handling request - aiohttp.server - 400, message="Bad status line 'Expected CRLF after version'" from cameras with bad HTTP implementations

[MrRiKo88]

The problem

After upgrading from 2023.4.6 to 2023.5.2 I started getting this error in the logs:

Logger: aiohttp.server Source: /usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:403 First occurred: 11:28:03 (41193 occurrences) Last logged: 14:44:44 Error handling request

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 332, in data_received messages, upgraded, tail = self._request_parser.feed_data(data) File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Expected CRLF after version'"

At the moment eveything seems to be working fine but the number of occurrences of the error keeps increasing significantly every time I check the logs.

What version of Home Assistant Core has the issue?

core-2023.5.2

What was the last working version of Home Assistant Core?

core-2023.4.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[HenJou]

I also get the same errors in my logs the only difference is

What type of installation are you running? Docker on Pi4

[MrRiKo88]

I also get the same errors in my logs the only difference is

What type of installation are you running? Docker on Pi4

I'm on HAOS in a VM on Proxmox

[bdraco]

Do you have a camera with older hikvision (or one of their subs) firmware on your network?

[HenJou]

Hi @bdraco Yes I have 2 hikvision dvr the one is DS-7216HGHI-E1 firmware V3.4.1 build 160325 and the config for every camera is as follow platform: generic name: "Kombuis" still_image_url: "http://192.168.xx.xx:8080/ISAPI/Streaming/channels/802/picture" stream_source: "rtsp://user:password@192.168.65.10:554/Streaming/Channels/801" verify_ssl: false username: !secret hik_username password: !secret hik_password authentication: digest

the 2nd one is DS-7216HQHI-K2 firmware V4.21.100 build 200307 using the ONVIF integration

If I can help to debug let me know

[bdraco]

Its a bug in the older hikvision firmwares. If there is a new firmware for the camera, updating will likely make the problem go away.

[HenJou]

@bdraco Did the upgrade on one dvr the other have no newer firmware and still no luck change my logger logs to aiohttp.server: fatal to ignore the error. Is their any else to do? Thanks

[MrRiKo88]

Do you have a camera with older hikvision (or one of their subs) firmware on your network?

Thanks for the suggestion..I have 6 Hikvision cameras with ONVIF integration. I'll try to upgrade the firmwares and see if something change.

[haforme]

Same issue with a Trendnet TV-IP420P using the ONVIF integration. It is a the latest firmware.
Home Assistant 2023.5.2 Supervisor 2023.04.1 Operating System 10.0 Frontend 20230503.3 - latest

Here's what I see in logs. Logger: aiohttp.server Source: /usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:403 First occurred: May 10, 2023 at 7:35:26 PM (7033 occurrences) Last logged: 12:21:28 PM

Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 332, in data_received messages, upgraded, tail = self._request_parser.feed_data(data) File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data aiohttp.http_exceptions.BadStatusLine: 400, message="Bad status line 'Expected CRLF after version'"

[home-assistant[bot]]

Hey there @hunterjm, mind taking a look at this issue as it has been labeled with an integration (onvif) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `onvif` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign onvif` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

onvif documentation onvif source _{^{(message by IssueLinks)}}

[bdraco]

There isn't much we can do about this as the problem is the camera is sending invalid http headers

[dgomes]

Unfortunately some cameras don't get firmware upgrades anymore, cameras still good and working in private networks.

Any chance of catching the exception and making it quiet ?

[bdraco]

The exception is happening in the header parser of aiohttp. It’s too deep in the stack to be able to do catch it without monkey patching aiohttp

[bdraco]

The parser aiohttp uses is https://github.com/nodejs/llhttp

[bdraco]

I think the only solution we could offer is the ability to disable webhooks on cameras, but that puts us in feature request territory so I won't discuss it on github as it belongs in https://community.home-assistant.io/c/feature-requests/13

[reotto613]

I have a DS-HD1 (Doorbell Cam) on Firmware V5.2.4 build 211101 which is the latest per this page: https://us.hikvision.com/en/products/more-products/video-intercom/door-station/3-mp-outdoor-wi-fi-smart-doorbell-camera-ds-hd1

I am still encountering this issue.

[bdraco]

Please reach out to the vendor and ask them to make their HTTP stack comply with the spec.

[bdraco]

HTTP request lines need to be separated with a CRLF and not an LF or CR

[bdraco]

Also Hikvision should know how to solve it as they have already fixed this exact problem in their other models

[bdraco]

Some of their older models had trailing white space in their http requests which can also trigger this problem

[bdraco]

Also feature request here if you want to vote for it https://community.home-assistant.io/t/disable-webhooks-on-cameras-that-have-old-firmwares-after-2023-5-x-some-old-onvif-cams-stopped-working/571854

[HenJou]

@MrRiKo88 I have added the following in my configuration.yaml file

logger: default: error logs: aiohttp.server: fatal

[MrRiKo88]

@MrRiKo88 I have added the following in my configuration.yaml file

logger: default: error logs: aiohttp.server: fatal

Many thanks!!

[craigmcgowan]

Good to see a fix to disable webhooks released in beta 2023.06. My log file has been impossible to read all month with so much spam from this error message. Currently it's reporting

First occurred: 29 May 2023 at 21:22:54 (712678 occurrences)

So about 10 million occurrences in the month!!

[bdraco]

The option to disable webhooks is available in 2023.6.x in the options flow for the config entry.