[volvooncall] offline communication error

[JungeG]

I upgraded to openhab 2.5.9 on Raspberry 4B using openhabian. The 2.5.9 volvooncall binding produces following error:

VolvoOnCall API

This bridge represents the gateway to Volvo On Call API.

Status: OFFLINE - COMMUNICATION_ERROR java.io.IOException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@b6e8d4::DecryptedEndPoint@8f4e0a{vocapi.wirelesscar.net/170.102.194.8:443<->/192.168.185.49:43284,OPEN,fill=-,flush=-,to=18/0}

Username and password are correct. Since the bridge is not only also the vehicle is not configured.

deinstalled and reinstalled the binding multiple times - no change. Thanks for any help

Georg

[wollyd]

I am also on 2.5.9 on a PI3 and have the same error on the bridge, this has been working on 2.5.8 but I have seen these kind of errors also on 2.5.8

Willy

[clinique]

I face the same with OH3. I think it may be a change on voc server side. Will have to investigate this.

Edit : may not be server side, on 2.5.7 it works. I think this may be related to some changes in Jetty. But did not found much as of now.

[Hilbrand]

All 2.5.x versions use the same jetty version because core did not change. So that might not explain why it does work and doesn't on some 2.5.x versions. To check this. What version of jetty is installed? And what version of Java is used?

[clinique]

In production, that works :

sysadmin@neo:~$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (Zulu 8.46.0.19-CA-linux64) (build 1.8.0_252-b14)
OpenJDK 64-Bit Server VM (Zulu 8.46.0.19-CA-linux64) (build 25.252-b14, mixed mode)

and core bundles are signed 2.5.0 (jetty does not appear with a bundle:list command), but Netty (is it the same ?) v 4.1.42

[clinique]

@JungeG : I've got a solution for this in v3.0, but I'm unable to find the issue origin on v2.9. So as of today, not backward compatible. Could you try with the v2.5.7 version of the binding ? On my prod environment, this works (also solving issue #8518)

[andifri]

@clinique: I have the same error in my v2.5.9 installation. I just did a quick and dirty downgrade of this addon to 2.5.7. It still shows the same error as before.

BTW, as you have solved this for v3.0, what is causing this error and what did you do to solve it in v3.0?

[clinique]

BTW, as you have solved this for v3.0, what is causing this error and what did you do to solve it in v3.0?

I stopped using Jetty http client, replace by native java http client, but this only works with JDK >= 11.

[JungeG]

@clinique: sorry for my late answer - I was busy over the weekend:

my java version:

java -version openjdk version "1.8.0_265" OpenJDK Runtime Environment (Zulu 8.48.3.246-CA-linux_aarch32hf) (build 1.8.0_265-b11) OpenJDK Client VM (Zulu 8.48.3.246-CA-linux_aarch32hf) (build 25.265-b11, mixed mode)

Using 2.5.7 I get the same error:

Status: OFFLINE - COMMUNICATION_ERROR java.io.IOException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@1390f4b::DecryptedEndPoint@16eb673{vocapi.wirelesscar.net/170.102.194.8:443<->/192.168.185.49:52368,OPEN,fill=-,flush=-,to=21/0}

Does the problem lay with the Zulu version?

I have seen that you made same corrections on OH3.0 version of the binding - can i use this with the openhab 2.5.9 core version?

Thanks for help Georg

[clinique]

I did not find where the problem resides. Now, on my production environment (2.5.7), I have the exact same issue than you. The solution I brought on v3.0 is not portable (in source code) to v2.5.9 and you will not be able to use v3.0 binding in a v2.5.x environment. So, sorry but as of today, no solution on this except moving as soon as possible OH3.

[JungeG]

I am running openhabian - how could I move to OH3 without loosing my configuration? Is there already a menu path to install OH3?

[clinique]

Don't know. Official OH3 version is expected in some weeks.

[JungeG]

@mstormi: Is there already an upgrade path to OH3? Thanks Georg

[clinique]

U can go to community forum, there are already some informations published regarding OH3

[andifri]

@clinique : Sorry, life got in the way of tinkering with OpenHAB.

Thank you for your input. I have the same java version (Zulu) as @JungeG. Reading your comments we will have to wait until OH3 is ready for prime time. Or at least I will wait as I am relying quite a lot on OH.

[mbronk]

For what it's worth - I am seeing the same issue on OH 2.5.9 running on Zulu 8.48.3.246-CA-linux_aarch32hf) (build 1.8.0_265-b11). Downgrading to OH 2.5.7 does not change the result either. Jetty version on 2.5.9 seems to be 9.4.20.v20190813 (this is based on existence of a JAR with that name in /opt/openhab2/runtime/system/org/eclipse/jetty/jetty-client - did not bother with any actual runtime check)

---

As a wild guess, given that - per @clinique - it does work on OH3 - I'd speculate it may have sth to do with the ciphersuite support in Jetty and/or JVM itself (though the 'new' Zulu releases of Java8 claim to support modern algorithms and TLS1.3...). Maybe, just maybe, related to https://github.com/eclipse/jetty.project/issues/3891 (?)

I was not able to spend much time on it, but when I was looking through the verbose logs, I got the impression the SSL handshake does go through (and negotiates TLSv1.2/TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384), though then there is an early EOF (decryption failure?) from the server-side... Unfortunately I have accidentally removed the original log so am unable to follow up on it :(

If anyone is interested in pursuing it further, I the next step might be to enable jetty debugging and SSL handshake troubleshooting via:

1. Adding -Djavax.net.debug=ssl,handshake (or '=all') to JAVA_OPTS (e.g. via /etc/default/openhab2)
2. Adding

   log4j2.logger.jetty.name = org.eclipse.jetty
   log4j2.logger.jetty.level = DEBUG

   to /opt/openhab2/userdata/etc/org.ops4j.pax.logging.conf

Hope it helps... Mateusz

[clinique]

Thanks, I landed at the same conclusion than jetty issue than you and spent soooooo much time trying to figure what was wrong under the hood. I will not try to make it 2.9 compatible as I'm switching dev now to focus on OH3. So if somebody is willing to work on v2.9.10 version, feel free to do so.

[mbronk]

UPDATE: I re-captured the SSL log, and everything looked clean there (handshake goes through, then the server closes the socket immediately after getting a HTTP request)... which got me thinking this may indeed be a server-side change and some kind of filtering...

Well... turns out it is (and JVM and Jetty are perfecly innocent here).

Turns out, VOC Server explicitly bans 'User-agent: Jetty/9.4.20.v20190813' header!

---

To confirm, see this:

$> curl -H "x-os-type: Android" -H "x-os-version: 22" -H "x-originator-type: App"  \
https://vocapi.wirelesscar.net/customerapi/rest/v3.0/customeraccounts \
--user "user@email.com:fake_password"

{"errorLabel":"CustomerAccountDoesNotExist","errorDescription":"An account with the supplied username does not exist."}

vs.

$>curl -A "Jetty/9.4.20.v20190813" \
-H "x-os-type: Android" -H "x-os-version: 22" -H "x-originator-type: App"  \
https://vocapi.wirelesscar.net/customerapi/rest/v3.0/customeraccounts \
--user "user@email.com:fake_password"

curl: (52) Empty reply from server

(note any other User-Agent header value - even using Jetty/9.4.20.v20190814 allows the request through)

---

@clinique - Is the VOC API in the binding coming from reverse-engineering the Android app, or is it public/documented somewhere? By the looks of it... it seems Volvo attempted ( not overly successfully :) ) to explicitly ban your binding (I'd assume it was generating a substantial traffic volume on their part, eg. due to retry attempts, failed logins etc.). Which leads me to a couple of conclusions:

1. The fact that it still works on OH3 is... because Volvo did not keep up with your change to native Java client. I guess they will get there some day though... :)
2. If you have a good relationship with VOC devs - it might be worthwhile to check what really caused them to blacklist the OH binding (if that was the case), so that we can be good API citizens and avoid it in the future...
3. Might be worth checking how much traffic can the VOC binding issue against a VOC server, so that we're simmilar to the 'app' usage and they do not see us as a threat. If we're perceived as a rogue client and they will want us off their platform - I guarantee they can make it happen via smarter filtering and app updates...
4. For now - I think it would be worthwhile to change the user-agent header on 2.5.x version of the binding to something else (should be doable from client code I presume)... And for the long run, maybe we should issue traffic exactly as the VOC app does (User-Agent header included(?))

[clinique]

@mbronk. I've not found any publically documented information regarding the VOC API. This has been built using existing source on the web and already present projects (e.g. molobrakoss's python lib) and reverse-engineering of the data flow. So...I have no special relationship with VOC devs. I think the binding could be improved, especially by caching results - thought of this recently because we probably query too often especially on item / channel linking that generates updates of the data. So I'm going to try getting back to Jetty in the current PR, modifying the user agent and restricting api requests as much as possible. If this is successfull, it will be possible to backport this to 2.9.10

[clinique]

@mbronk : just made the test switching back to Jetty client and overrinding the user-agent, still the same result 😒

[mbronk]

@clinique - That's interesting. Have you tried the curl examples I attached? I am still seing a consistent behavior there: EOF when using Jetty/9.4.20.v20190813 as User-Agent, and a valid response - otherwise. I can also replicate this via Postman...

Are you sure the user-agent override is in effect? Can you perhaps check what is sent on the wire? (either by using -Djavax.net.debug=all added to JVM params or by some packet sniffer like Fiddler or Wireshark?). It might be a different issue, but my bet is stil on the User-Agent (assuming you're getting the same EOF I am)

[clinique]

Are you sure the user-agent override is in effect? Can you perhaps check what is sent on the wire? (either by using -Djavax.net.debug=all added to JVM params or by some packet sniffer like Fiddler or Wireshark?). It might be a different issue, but my bet is stil on the User-Agent (assuming you're getting the same EOF I am)

No, that's what I must double check.

[clinique]

@clinique - That's interesting. Have you tried the curl examples I attached? I am still seing a consistent behavior there: EOF when using Jetty/9.4.20.v20190813 as User-Agent, and a valid response - otherwise. I can also replicate this via Postman...

Are you sure the user-agent override is in effect? Can you perhaps check what is sent on the wire? (either by using -Djavax.net.debug=all added to JVM params or by some packet sniffer like Fiddler or Wireshark?). It might be a different issue, but my bet is stil on the User-Agent (assuming you're getting the same EOF I am)

It works !

[clinique]

@mbronk : thanks for having pointed me in the right direction. PR pushed, brought back to Jetty, so potentially backportable to OH2.9. Binding enhanced with a cache mechanism.

[JungeG]

So - what do I have to do? Just wait for the update of the binding - or do I have to install an other version of thee binding? Thanks for a quick reply. Georg

[clinique]

@JungeG : the PR is under review and could be merged in the coming days. Can not predict exactly when.

[mbronk]

@clinique - what do you think would it take to backport just this change: httpClient.setUserAgentField(new HttpField(HttpHeader.USER_AGENT, "openhab/voc_binding/" + InstanceUUID.get())); to the 2.5.x branch...? Do you think UserAgent change can be done while retaining HttpClient helper class, but without changes in core (org.openhab.core.io.net.http::HttpUtil) ?

In particular, if you provide 'User-Agent' as a header via the generic 'header' API... is it in effect, or does one need to use the .setUserAgentField() method? If the latter - can we subclass HttpUtil and provide it with a custom HttpClientFactory doing the same?

Unfortunately, I do not have any devenv setup as of now, so unable to check in runtime... though my hope is that someone who has it, would be able to fairly easily backport your fix to 2.5.x, which would accomplish 2 things at once:

• Intercept 2.5.10 (if it gets released)
• Make Travis build it, so people could download the JAR directly from CI and replace the bundled binding version when on 2.5.9

[clinique]

@clinique - what do you think would it take to backport just this change: httpClient.setUserAgentField(new HttpField(HttpHeader.USER_AGENT, "openhab/voc_binding/" + InstanceUUID.get())); to the 2.5.x branch...? Do you think UserAgent change can be done while retaining HttpClient helper class, but without changes in core (org.openhab.core.io.net.http::HttpUtil) ?

This is the change that has to be back-ported and current version (2.5.9) of the binding does not use directly HttpClient but instead delegates it to HpptUtil, so most (or all) of the changes of the current PR have to be backported.

In particular, if you provide 'User-Agent' as a header via the generic 'header' API... is it in effect, or does one need to use the .setUserAgentField() method? If the latter - can we subclass HttpUtil and provide it with a custom HttpClientFactory doing the same?

No, I made the test, keeping HttpUtil and adding user-agent in the header, this does not work.

Unfortunately, I do not have any devenv setup as of now, so unable to check in runtime... though my hope is that someone who has it, would be able to fairly easily backport your fix to 2.5.x, which would accomplish 2 things at once:

* Intercept 2.5.10 (if it gets released)

* Make Travis build it, so people could download the JAR directly from CI and replace the bundled binding version when on 2.5.9

openhab2-addons repo is now read-only, so I think someone has to back-port the PR in local, generate the JAR and make it available on a public share for those facing (I suppose everybody) this issue.

[mbronk]

@clinique - I had some time today and backported the fixes to 2.5.x branch. On my setup it seems to compile and work fine... Please take a look (PR link above)

[mbronk]

@JungeG, @wollyd, @andifri (&others) - Just so you know - once PR #8660 is merged, a fix for this issue should intercept OpenHAB 2.5.10 (of course, it will be in 3.0 as well)

If you're interested in fixing it in the current 2.5.x version you have, you can download the new version of the binding with the fix here: https://openhab.jfrog.io/openhab/libs-pullrequest-local/org/openhab/addons/bundles/org.openhab.binding.volvooncall/2.5.10-SNAPSHOT/org.openhab.binding.volvooncall-2.5.10-SNAPSHOT.jar

You will need to uninstall the stock binding (e.g. via PaperUI), and drop the JAR file into the /addons folder in your OH installation directory.

Hope it helps...

[JungeG]

@mbronk : Thanks, I have installed the 2.5.10 version today and it works without any Problem. Kind regards Georg