home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.07k stars 29.73k forks source link

Invalid ONVIF SubscriptionManager RenewRequest - python-zeep WSA bug #45513

Closed slydiman closed 1 year ago

slydiman commented 3 years ago

The problem

The problem is missing addressing in the SubscriptionManager/RenewRequest. The Header contains only the Security element. Action, To and MessageID are missing in the Header of the Renew request. As the result a lot of ONVIF cameras (XiongMai, Wanscam and other) response 400 (Bad Request) to RenewRequest. It is impossible to add/setup these cameras at all. You can see different errors in logs, but the reason is the same. My patch to python-zeep which fixed this issue was reverted. So it must be fixed in Home Assistant code.

Environment

Problem-relevant configuration.yaml

logger:
  default: info
  logs:
    homeassistant.components.onvif: debug

Traceback/Error logs

2021-01-25 03:57:41 DEBUG (MainThread) [homeassistant.components.onvif] Setting up the ONVIF device management service
2021-01-25 03:57:41 DEBUG (MainThread) [homeassistant.components.onvif] Retrieving current device date/time
2021-01-25 03:57:41 DEBUG (MainThread) [homeassistant.components.onvif] Device date/time: 2021-01-24 23:57:42+00:00 | System date/time: 2021-01-24 23:57:41.482773+00:00
2021-01-25 03:57:44 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Cam1 - xx:xx:xx:xx:xx:xx for onvif
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 236, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/usr/src/homeassistant/homeassistant/components/onvif/__init__.py", line 77, in async_setup_entry
    if not await device.async_setup():
  File "/usr/src/homeassistant/homeassistant/components/onvif/device.py", line 105, in async_setup
    self.capabilities = await self.async_get_capabilities()
  File "/usr/src/homeassistant/homeassistant/components/onvif/device.py", line 252, in async_get_capabilities
    pullpoint = await self.events.async_start()
  File "/usr/src/homeassistant/homeassistant/components/onvif/event.py", line 83, in async_start
    await self.async_renew()
  File "/usr/src/homeassistant/homeassistant/components/onvif/event.py", line 154, in async_renew
    await self._subscription.Renew(termination_time)
  File "/usr/local/lib/python3.8/site-packages/zeep/proxy.py", line 64, in __call__
    return await self._proxy._binding.send_async(
  File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 164, in send_async
    return self.process_reply(client, operation_obj, response)
  File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 229, in process_reply
    return self.process_error(doc, operation)
  File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 381, in process_error
    subcode_qname = as_qname(
  File "/usr/local/lib/python3.8/site-packages/zeep/utils.py", line 36, in as_qname
    raise XMLParseError("No namespace defined for %r (%r)" % (prefix, value))
zeep.exceptions.XMLParseError: No namespace defined for 'ter' ('ter:InvalidArgVal')

Additional information

probot-home-assistant[bot] commented 3 years ago

onvif documentation onvif source (message by IssueLinks)

slydiman commented 3 years ago

Here is the problem request:

POST /onvif/events_service HTTP/1.1
Host: 192.168.3.208
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
User-Agent: Zeep/4.0.0 (www.python-zeep.org)
SOAPAction: "http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/RenewRequest"
Content-Type: application/soap+xml; charset=utf-8; action="http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/RenewRequest"
Content-Length: 1007

<?xml version='1.0' encoding='utf-8'?>
<soap-env:Envelope xmlns:soap-env="http://www.w3.org/2003/05/soap-envelope">
  <soap-env:Header>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UsernameToken>
        <wsse:Username>admin</wsse:Username>
        <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">XXX</wsse:Password>
        <wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">ZZZ</wsse:Nonce>
        <wsu:Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2021-03-26T14:35:28+00:00</wsu:Created>
      </wsse:UsernameToken>
    </wsse:Security>
  </soap-env:Header>
  <soap-env:Body>
    <ns0:Renew xmlns:ns0="http://docs.oasis-open.org/wsn/b-2">
      <ns0:TerminationTime>2021-03-27T14:35:28Z</ns0:TerminationTime>
    </ns0:Renew>
  </soap-env:Body>
</soap-env:Envelope>

and the camera response

HTTP/1.1 400 Bad Request
Server: gSOAP/2.8
Content-Type: application/soap+xml; charset=utf-8; action="http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/RenewRequest"
Content-length: 2012
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://www.w3.org/2003/05/soap-envelope" xmlns:SOAP-ENC="http://www.w3.org/2003/05/soap-encoding" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:wsdd="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:chan="http://schemas.microsoft.com/ws/2005/02/duplex" xmlns:wsa5="http://www.w3.org/2005/08/addressing" xmlns:c14n="http://www.w3.org/2001/10/xml-exc-c14n#" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xenc="http://www.w3.org/2001/04/xmlenc#" xmlns:wsc="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:xmime="http://tempuri.org/xmime.xsd" xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:tt="http://www.onvif.org/ver10/schema" xmlns:wsrfbf="http://docs.oasis-open.org/wsrf/bf-2" xmlns:wstop="http://docs.oasis-open.org/wsn/t-1" xmlns:wsrfr="http://docs.oasis-open.org/wsrf/r-2" xmlns:tdn="http://www.onvif.org/ver10/network/wsdl" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:tdv="http://www.onvif.org/ver10/events/wsdl" xmlns:wsnt="http://docs.oasis-open.org/wsn/b-2" xmlns:tptz="http://www.onvif.org/ver20/ptz/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" xmlns:tan="http://www.onvif.org/ver20/analytics/wsdl" xmlns:timg="http://www.onvif.org/ver20/imaging/wsdl">
  <SOAP-ENV:Body>
    <SOAP-ENV:Fault>
      <SOAP-ENV:Code>
        <SOAP-ENV:Value>SOAP-ENV:Receiver</SOAP-ENV:Value>
        <SOAP-ENV:Subcode>
          <SOAP-ENV:Value>ter:InvalidArgVal</SOAP-ENV:Value>
          <SOAP-ENV:Subcode>
            <SOAP-ENV:Value>ter:ModeError</SOAP-ENV:Value>
          </SOAP-ENV:Subcode>
        </SOAP-ENV:Subcode>
      </SOAP-ENV:Code>
      <SOAP-ENV:Reason>
        <SOAP-ENV:Text xml:lang="en">
      </SOAP-ENV:Reason>
    </SOAP-ENV:Fault>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

I have used ONVIF Device Manager v2.2.250 and got the following correct communicating with the same camera (Wanscam K21): Request:

POST /onvif/events_service HTTP/1.1
Content-Type: application/soap+xml; charset=utf-8
Host: 192.168.3.208
Content-Length: 1339
Accept-Encoding: gzip, deflate
Connection: Close

<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing">
  <s:Header>
    <a:Action s:mustUnderstand="1">http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/RenewRequest</a:Action>
    <a:MessageID>urn:uuid:57c83690-ef2c-4817-aba7-f56b42f3853e</a:MessageID>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
    <Security s:mustUnderstand="1" xmlns="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <UsernameToken>
        <Username>admin</Username>
        <Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">XXX</Password>
        <Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">ZZZ</Nonce>
        <Created xmlns="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2021-03-26T15:34:58.255Z</Created>
      </UsernameToken>
    </Security>
    <a:To s:mustUnderstand="1">http://192.168.3.208/onvif/events_service</a:To>
  </s:Header>
  <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <Renew xmlns="http://docs.oasis-open.org/wsn/b-2">
      <TerminationTime>PT1M</TerminationTime>
    </Renew>
  </s:Body>
</s:Envelope>

and response:

HTTP/1.1 200 OK
Server: gSOAP/2.8
Content-Type: application/soap+xml; charset=utf-8
Content-length: 827
Connection: close

<?xml version="1.0" encoding="utf-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://www.w3.org/2003/05/soap-envelope" xmlns:SOAP-ENC="http://www.w3.org/2003/05/soap-encoding" xmlns:wsa5="http://www.w3.org/2005/08/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsnt="http://docs.oasis-open.org/wsn/b-2" xmlns:tev="http://www.onvif.org/ver10/events/wsdl">
  <wsa5:To SOAP-ENV:mustUnderstand="true">http://192.168.3.208:80/onvif/e</wsa5:To>
  </SOAP-ENV:Header>
  <SOAP-ENV:Body>
    <wsnt:RenewResponse>
      <wsnt:CurrentTime>2021-03-26T19:34:58Z</wsnt:CurrentTime>
      <wsnt:TerminationTime>2021-03-26T19:35:58Z</wsnt:TerminationTime>
    </wsnt:RenewResponse>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
slydiman commented 3 years ago

It seems SubscriptionManager/RenewRequest is totally broken. I can add other camera XiongMai A5-X20RJ (IPC_XM530_RA50X20) and it works. But I see the same response (400 Bad Request) to RenewRequest

<?xml version='1.0' encoding='utf-8'?>
<soap-env:Envelope xmlns:soap-env="http://www.w3.org/2003/05/soap-envelope">
  <soap-env:Header>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UsernameToken>
        <wsse:Username>admin</wsse:Username>
        <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">JQpvpHyTidtpWczlmA5mVT5F+hA=</wsse:Password>
        <wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">VHksOlqcRXaawiywHUWIVg==</wsse:Nonce>
        <wsu:Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2021-03-30T09:46:02+00:00</wsu:Created>
      </wsse:UsernameToken>
    </wsse:Security>
  </soap-env:Header>
  <soap-env:Body>
    <ns0:Renew xmlns:ns0="http://docs.oasis-open.org/wsn/b-2">
      <ns0:TerminationTime>2021-03-31T09:46:02Z</ns0:TerminationTime>
    </ns0:Renew>
  </soap-env:Body>
</soap-env:Envelope>

Response:

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:e="http://www.w3.org/2003/05/soap-encoding" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl" xmlns:wsnt="http://docs.oasis-open.org/wsn/b-2" xmlns:wstop="http://docs.oasis-open.org/wsn/t-1" xmlns:wsntw="http://docs.oasis-open.org/wsn/bw-2" xmlns:wsrf-rw="http://docs.oasis-open.org/wsrf/rw-2" xmlns:wsrf-r="http://docs.oasis-open.org/wsrf/r-2" xmlns:wsrf-bf="http://docs.oasis-open.org/wsrf/bf-2" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl" xmlns:wsoap12="http://schemas.xmlsoap.org/wsdl/soap12" xmlns:http="http://schemas.xmlsoap.org/wsdl/http" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:wsadis="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:tt="http://www.onvif.org/ver10/schema" xmlns:tns1="http://www.onvif.org/ver10/topics" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" xmlns:tev="http://www.onvif.org/ver10/events/wsdl" xmlns:timg="http://www.onvif.org/ver20/imaging/wsdl" xmlns:tst="http://www.onvif.org/ver10/storage/wsdl" xmlns:dn="http://www.onvif.org/ver10/network/wsdl" xmlns:tr2="http://www.onvif.org/ver20/media/wsdl" xmlns:tptz="http://www.onvif.org/ver20/ptz/wsdl" xmlns:tan="http://www.onvif.org/ver20/analytics/wsdl" xmlns:axt="http://www.onvif.org/ver20/analytics" xmlns:tmd="http://www.onvif.org/ver10/deviceIO/wsdl" xmlns:ter="http://www.onvif.org/ver10/error">
  <s:Header>
    <wsa:Action>http://www.w3.org/2005/08/addressing/soap/fault</wsa:Action>
  </s:Header>
  <s:Body>
    <s:Fault>
      <s:Code>
        <s:Value>s:Receiver</s:Value>
        <s:Subcode>
          <s:Value>wsrf-rw:ResourceUnknownFault</s:Value>
        </s:Subcode>
      </s:Code>
      <s:Reason>
        <s:Text xml:lang="en">ResourceUnknownFault</s:Text>
      </s:Reason>
    </s:Fault>
  </s:Body>
</s:Envelope>

As the result I don't see any events. I expected the motion alarm/detection at least. But I don't see any sensors at all.

slydiman commented 3 years ago

I have partially fixed the issue removing the line https://github.com/home-assistant/core/blob/b58d6a6293ef3f5bd537559516997858abcabcac/homeassistant/components/onvif/event.py#L86

It is necessary to initialize events first before renew anything. After that

But the problem returns after 1 minute trying to renew subscription. It seems the problem is missing addressing in the Renew request. The Header contains only Security element. Action, To and MessageID are missing in the Header of Renew request. Currently I have no idea how to fix it.

Hope someone of code owners can look at this issue.

slydiman commented 3 years ago

This patch must fix the issue https://github.com/mvantellingen/python-zeep/pull/1206

gomble commented 3 years ago

https://github.com/home-assistant/core/pull/48571

what does this mean ? how to fix this now?

gomble commented 3 years ago

https://github.com/home-assistant/core/pull/42651/files

solved this with editing and reverting back the code

jrb80 commented 3 years ago

@slydiman is this issue now patched in an upcoming release?

gomble commented 3 years ago

@slydiman is this issue now patched in an upcoming release?

Code owner replied no

slydiman commented 3 years ago

@slydiman is this issue now patched in an upcoming release?

https://github.com/slydiman/ha_custom_onvif Any comments/improvements are welcome.

gomble commented 3 years ago

@slydiman is this issue now patched in an upcoming release?

https://github.com/slydiman/ha_custom_onvif Any comments/improvements are welcome.

worked for me wow thanks!

sag3ll0 commented 3 years ago

@slydiman is this issue now patched in an upcoming release?

https://github.com/slydiman/ha_custom_onvif Any comments/improvements are welcome.

its work for me. Thanks

Enjoy-Combi commented 3 years ago

Hi,

Thanks a lot for your work ! I have now a lot of sensors for a XM530_R80X30-PQ_8M: Capture dโ€™eฬcran 2021-05-22 aฬ€ 07 35 07

I am new with onvif, regarding PTZ, is it something to manage separately to the ONVIF integration ? It would be wonderful to have this automatically configured when add the cam. Like it has been successfully carried out with the app onvifer.

slydiman commented 3 years ago

regarding PTZ, is it something to manage separately to the ONVIF integration ?

XM530_R80X30-PQ_8M does not store PTZ presets. You can add some PTZ presets to the camera using ONVIF Device Manager, then restart HA. Now you can use onvif.ptz service with presets PRESET_0, PRESET_1, etc., but only till reboot the camera.

  - service: onvif.ptz
    data:
      entity_id: camera.cam1_hd
      move_mode: GotoPreset
      preset: "PRESET_0"
Enjoy-Combi commented 3 years ago

Many thanks,

Will try soon. It remains strange for me that some applications manage this automatically. Will keep you informed.

Enjoy-Combi commented 3 years ago

Hi,

Back on the configuration, I was able to define some presets but I don't know where to add the data :

Are the presets needed to move / zoom ? I wonder not but.

Enjoy-Combi commented 3 years ago

Found, have to add data into the panel.

This example creates buttons to move the camera: https://community.home-assistant.io/t/onvif-camera-platform-how-to-do-ptz/197447/4

Then all good, many thanks for your good job ! and still don't understand why the onvif integration is not taking benefit of your work.

sag3ll0 commented 3 years ago

Found, have to add data into the panel.

This example creates buttons to move the camera: https://community.home-assistant.io/t/onvif-camera-platform-how-to-do-ptz/197447/4

Then all good, many thanks for your good job ! and still don't understand why the onvif integration is not taking benefit of your work.

Thanks, its working for me/

slydiman commented 3 years ago

This patch must fix the issue mvantellingen/python-zeep#1206

Zeep 4.1.0 has been released yesterday. This patch was merged and reverted immediately with the following comment:

Just reverted it again, needs more thinking (if you always want WSA you could just list it in the plugins)

It means that this issue must be fixed in ONVIF integration which is a part of Home Assistant Core.

Enjoy-Combi commented 2 years ago

Hi,

Thanks for the information regarding the merge of the patch. Does it mean we can remove the personal integration right now ?

Pan & tilt work good. But impossible to get zoom_in & zoom_out to work. Here is what I have in my configuration: - entity: camera.nvt_mainstream tap_action: action: call-service service: onvif.ptz service_data: entity_id: camera.nvt_mainstream zoom: ZOOM_OUT distance: 1 move_mode: ContinuousMove continuous_duration: 1

In the log, there is nothing related to onvif, camera.

slydiman commented 2 years ago

Thanks for the information regarding the merge of the patch. Does it mean we can remove the personal integration right now ?

Do you mean my custom ONVIF integration? The patch to Zeep was not applied finally. Nothing changed.

Enjoy-Combi commented 2 years ago

Exactly, I was referring to your custom ONVIF integration. Thank you for your confirmation.

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment ๐Ÿ‘ This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

slydiman commented 2 years ago

Ping

Enjoy-Combi commented 2 years ago

Hello,

I have got a new camera : EYEPLUS_DEV from EYEPLUS Firmware: 5.6.1.1127

I have the main & sub stream but no sensors. How to find / add them to home assistant ?

Many thanks

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment ๐Ÿ‘ This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

slydiman commented 2 years ago

Ping

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment ๐Ÿ‘ This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

slydiman commented 2 years ago

Ping

new-kirte commented 1 year ago

bump

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment ๐Ÿ‘ This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

slydiman commented 1 year ago

Ping

bdraco commented 1 year ago

This patch must fix the issue

mvantellingen/python-zeep#1206

Zeep 4.1.0 has been released yesterday. This patch was merged and reverted immediately with the following comment:

Just reverted it again, needs more thinking (if you always want WSA you could just list it in the plugins)

It means that this issue must be fixed in ONVIF integration which is a part of Home Assistant Core.

Did you ever open a PR to core?

slydiman commented 1 year ago

Did you ever open a PR to core?

No. Zeep is not a part of HA Core. I made the patch to Zeep. But I have no idea how to configure plugins for Zeep. Probably it must be fixed in onvif-zeep-async which is not a part of HA Core too. Note Onvif integration from HA Core requires onvif-zeep-async 1.2.2