search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.69k stars 30.44k forks source link

CamHipro Wireless WiFi 5MP 240X ZOOM Camera cannot parse XML or respond to `GetCapabilities` with 2023.5.x and later #92308

Closed criticallimit closed 1 year ago

criticallimit commented 1 year ago

The problem

next onvif problem here

What version of Home Assistant Core has the issue?

2023.5.0b5

What was the last working version of Home Assistant Core?

2023.4.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

onvif

Link to integration documentation on our website

No response

Diagnostics information

Logger: homeassistant.config_entries Source: components/onvif/device.py:309 First occurred: 22:12:07 (1 occurrences) Last logged: 22:12:07

Error setting up entry Cam Zoom 3 - C4:3C:B0:EA:72:5A for onvif
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 387, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/usr/src/homeassistant/homeassistant/components/onvif/__init__.py", line 38, in async_setup_entry
    await device.async_setup()
  File "/usr/src/homeassistant/homeassistant/components/onvif/device.py", line 112, in async_setup
    self.profiles = await self.async_get_profiles()
  File "/usr/src/homeassistant/homeassistant/components/onvif/device.py", line 309, in async_get_profiles
    result = await media_service.GetProfiles()
  File "/usr/local/lib/python3.10/site-packages/zeep/proxy.py", line 64, in __call__
    return await self._proxy._binding.send_async(
  File "/usr/local/lib/python3.10/site-packages/zeep/wsdl/bindings/soap.py", line 156, in send_async
    response = await client.transport.post_xml(
  File "/usr/local/lib/python3.10/site-packages/zeep/transports.py", line 235, in post_xml
    response = await self.post(address, message, headers)
  File "/usr/local/lib/python3.10/site-packages/zeep/transports.py", line 220, in post
    response = await self.client.post(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1845, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 116, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 95, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 159, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 195, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError
pbvdven commented 1 year ago 
logger:
  default: info
  logs:
    homeassistant.components.onvif: debug
    httpx: debug
    onvif: debug
    zeep: debug

When i enable this i get info from all onvif devices its not easy for me to read the log to see what belongs to the camera with the issue im sorry. this belongs to the issue camera for sure its filtert on IP.

2023-05-06 17:16:46.490 DEBUG (MainThread) [homeassistant.components.onvif] Fetching profiles from ONVIF device {'host': '192.168.2.189', 2023-05-06 17:16:46.491 DEBUG (MainThread) [onvif] Creating service ('devicemgmt', None) with http://192.168.2.189:8080/onvif/device_service 2023-05-06 17:16:46.497 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/device_service: b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\nhttp://www.onvif.org/ver10/device/wsdl/GetCapabilities</wsa:Action>urn:uuid:2ad0b8eb-b568-4e4b-a7d2-9c99f178bc58</wsa:MessageID>http://192.168.2.189:8080/onvif/device_service</wsa:To></wsse:Username>=</wsse:Password>****==</wsse:Nonce>2023-05-06T15:16:46+00:00</wsu:Created></wsse:UsernameToken></wsse:Security></soap-env:Header>All</ns0:Category></ns0:GetCapabilities></soap-env:Body></soap-env:Envelope>'

bdraco commented 1 year ago

Can you share the full log?

My email is nick [at] koston [dot] org

bdraco commented 1 year ago 
2023-05-06 17:33:11.072 DEBUG (MainThread) [homeassistant.components.onvif] PTZ: Could not determine camera capabilities: Device sent empty error with type <class 'httpx.ReadTimeout'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 32, in read
    with anyio.fail_after(timeout):
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 116, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 95, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 159, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 195, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/onvif/config_flow.py", line 277, in async_setup_profiles
    await device.update_xaddrs()
  File "/usr/local/lib/python3.10/site-packages/onvif/client.py", line 527, in update_xaddrs
    capabilities = await devicemgmt.GetCapabilities({"Category": "All"})
  File "/usr/local/lib/python3.10/site-packages/zeep/proxy.py", line 64, in __call__
    return await self._proxy._binding.send_async(
  File "/usr/local/lib/python3.10/site-packages/zeep/wsdl/bindings/soap.py", line 156, in send_async
    response = await client.transport.post_xml(
  File "/usr/local/lib/python3.10/site-packages/zeep/transports.py", line 235, in post_xml
    response = await self.post(address, message, headers)
  File "/usr/local/lib/python3.10/site-packages/zeep/transports.py", line 220, in post
    response = await self.client.post(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1845, in post
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
bdraco commented 1 year ago

So the camera just doesn't respond to the GetCapabilities request.

bdraco commented 1 year ago

Its likely a subtle bug in the camera's XML parser that choking on something that changed between 2023.4.4 and 2023.5.x

I'll have to wait for one of the devices to show up to try to come up with a work around as it will take some experimenting as the XML we are sending is valid, but the camera failed to handle it correctly.

bdraco commented 1 year ago

@pbvdven Thanks for the logs. At least we have a better idea on whats going on now

bdraco commented 1 year ago

The only thing that changed with XML is the addition of the WSAs between those versions.

I made a PR to turn them back off for some of the services and left them on for the services that are reported to need them.

Can you try the following?

cd /config
curl -o- -sSL https://gist.githubusercontent.com/bdraco/43f8043cb04b9838383fd71353e99b18/raw/core_integration_pr | bash /dev/stdin -d onvif -p 92700
cat /config/custom_components/onvif/manifest.json

Verify that the version displayed above is "onvif-zeep-async==2.1.2" and NOT 2.1.1

Restart HA

Test

Delete /config/custom_components/onvif and restart after testing is completed to restore the core version

bdraco commented 1 year ago

Edit: there was a problem with the checkout script above. I edited the command and fixed it

pbvdven commented 1 year ago

this takes forever image

pbvdven commented 1 year ago

i have send you a new log.

I have the camera connected by ethernet cable. i read above that some models don't work over wifi so i thought i mention i'm not using wifi.

bdraco commented 1 year ago

It doesn't look like the changes are in place.

Can you run cat /config/custom_components/onvif/manifest.json and post the result?

pbvdven commented 1 year ago

image

image

image

pbvdven commented 1 year ago

also i said something like custom in the corner of the onvif integration

bdraco commented 1 year ago

It looks right. Digging through the log some more to figure out why the WSAs are still on

bdraco commented 1 year ago

o wait. It does look like its making more progress

bdraco commented 1 year ago

So its definitely the WSAs that cause the problem

bdraco commented 1 year ago

Thats really unfortunate since if I take them back out it will break hundreds of other cameras

pbvdven commented 1 year ago

that not so good news there is not something that you can do like when they dont get accepted or error you dont use them idk maybe its stupid. it worked really nice before mainly for the ptz controls on a webrtc card.

bdraco commented 1 year ago

The problem is the camera just doesn't respond at all when the WSA are sent. No errors, no nothing.

bdraco commented 1 year ago

We don't know if the camera has gone away or its just failing to respond because it does nothing when it gets a request with WSAs

pbvdven commented 1 year ago

i get it. is there maybe a option you can make one custom without the wsa's that i can use as a custom onvif integration? or can i use the last version from the onvif integration shipped with 2023.4 that one worked fine for all my 3 onvif devices the hikvision ds-hd1, dahua cube 4mp and my cheap ptz.

bdraco commented 1 year ago

I'm trying to work up an alternate solution. Give me ~30m

pbvdven commented 1 year ago

take all the time you need. thanks for looking in to it.

pbvdven commented 1 year ago

Idk if it could help but these onvif settings can be set in the camera. image

NVC type has also the option uniview

bdraco commented 1 year ago

I pushed onvif-zeep-async==2.1.4

cd /config
curl -o- -sSL https://gist.githubusercontent.com/bdraco/43f8043cb04b9838383fd71353e99b18/raw/core_integration_pr | bash /dev/stdin -d onvif -p 92700
cat /config/custom_components/onvif/manifest.json

Verify that the version displayed above is "onvif-zeep-async==2.1.4" and NOT 2.1.1

Restart HA

Test

Delete /config/custom_components/onvif and restart after testing is completed to restore the core version

pbvdven commented 1 year ago

i do get other labels image

pbvdven commented 1 year ago

same result new log in your mail

bdraco commented 1 year ago

no ReadError in the log so definitely making progress

bdraco commented 1 year ago

Can you try power cycling the device before setting it up again with the 2.1.4 version?

bdraco commented 1 year ago

Also the labels are missing because its a custom component and the translation files aren't there (this is normal)

pbvdven commented 1 year ago

now its added but still image

also new log

bdraco commented 1 year ago

I didn't get a new log via email. Not sure if it got stuck going out ?

pbvdven commented 1 year ago

said send here but i send it again

my bad send it through icloud.com and your verification email got into spam

bdraco commented 1 year ago

got it.

Its getting a lot further.

2023-05-07 00:02:51.389 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/device_service:
2023-05-07 00:02:51.400 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/device_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.401 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/device_service (status: 200):
2023-05-07 00:02:51.403 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/device_service:
2023-05-07 00:02:51.413 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/device_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.415 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/device_service (status: 200):
2023-05-07 00:02:51.416 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/device_service:
2023-05-07 00:02:51.425 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/device_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.426 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/device_service (status: 200):
2023-05-07 00:02:51.427 DEBUG (MainThread) [onvif] Creating service ('media', None) with http://192.168.2.189:8080/onvif/media_service
2023-05-07 00:02:51.429 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/media_service:
2023-05-07 00:02:51.434 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/media_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.435 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/media_service (status: 200):
2023-05-07 00:02:51.436 DEBUG (MainThread) [onvif] Creating service ('imaging', None) with http://192.168.2.189:8080/onvif/image_service
2023-05-07 00:02:51.436 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/media_service:
2023-05-07 00:02:51.473 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/media_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.475 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/media_service (status: 200):
2023-05-07 00:02:51.479 DEBUG (MainThread) [onvif] Creating service ('ptz', None) with http://192.168.2.189:8080/onvif/ptz_service
2023-05-07 00:02:51.607 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/ptz_service:
2023-05-07 00:02:51.624 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/ptz_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.631 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/ptz_service (status: 200):
2023-05-07 00:02:51.679 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/ptz_service:
2023-05-07 00:02:51.693 DEBUG (MainThread) [httpx] HTTP Request: POST http://192.168.2.189:8080/onvif/ptz_service "HTTP/1.1 200 OK"
2023-05-07 00:02:51.701 DEBUG (MainThread) [zeep.transports] HTTP Response from http://192.168.2.189:8080/onvif/ptz_service (status: 200):
2023-05-07 00:02:51.738 DEBUG (MainThread) [onvif] Creating service ('notification', None) with http://192.168.2.189:8080/onvif/event_service
2023-05-07 00:02:51.740 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/event_service:
2023-05-07 00:03:26.249 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/event_service:
2023-05-07 00:03:56.253 DEBUG (MainThread) [zeep.transports] HTTP Post to http://192.168.2.189:8080/onvif/media_service:
2023-05-07 00:04:26.258 WARNING (MainThread) [homeassistant.config_entries] Config entry 'PTZ - 00:0A:24:33:32:D2' for onvif integration not ready yet: Could not connect to camera 192.168.2.189:8080: ; Retrying in background
bdraco commented 1 year ago

Can you adjust the loggers so we can pick up the custom component and try again

# Example configuration.yaml entry
logger:
  default: info
  logs:
    homeassistant.components.onvif: debug
    custom_components.onvif: debug
    httpx: debug
    onvif: debug
    zeep: debug
bdraco commented 1 year ago

Also please power cycle the device before trying again as I think once it gets into a bad state it won't recover on its own

bdraco commented 1 year ago

The last one you just sent the camera didn't respond at all to the request. I'm not sure it was booted yet or was still in a the stuck state

pbvdven commented 1 year ago

new log now its added again.

bdraco commented 1 year ago

Thanks. Its back to completely not responding again. This could take a few days of iterations to figure out the root cause

bdraco commented 1 year ago

It might be better to wait until the test camera shows up

pbvdven commented 1 year ago

thanks for your time. Ive i can do anything else to help just let me know

bdraco commented 1 year ago

I’ll follow up when the hardware arrives

bdraco commented 1 year ago

Original camera I ordered shows in the US now so hopefully have it by next weekend

bdraco commented 1 year ago

Camera showed up. Turns out its likely the same issue as #91678

bdraco commented 1 year ago

Please try 2023.5.4 when it is released

pbvdven commented 1 year ago

Please try 2023.5.4 when it is released

Thanks with 2023.5.4 the 5mp ptz works again.