search

openhab / openhab-addons

Add-ons for openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.87k stars 3.58k forks source link

[linky] Timeout issue #7610

Closed Ozneb closed 3 years ago

Ozneb commented 4 years ago

I've implemented the linky addon with copy/paste of the sample configuration, i replace by my login/password and have a timeout issue in paperUI : "Status: OFFLINE - COMMUNICATION_ERROR timeout" At first, with just thing file without any items connected, the thing is Online, but with all the channels connected, i've got the timeout issue. I've tried to comment my items file but, without succes, the thing is always in error.

PS : I'm using openhab with the official 2.5.4 docker container (https://hub.docker.com/r/openhab/openhab/)

For the thing file: Thing linky:linky:local "Compteur Linky" [ username="example@domaine.fr", password="**" ]

For the items file: Number:Energy Linky_ConsoHier "Conso hier [%.0f %unit%]" { channel="linky:linky:local:daily#yesterday" } Number:Energy Linky_ConsoSemaineEnCours "Conso semaine en cours [%.0f %unit%]" { channel="linky:linky:local:weekly#thisWeek" } Number:Energy Linky_ConsoSemaineDerniere "Conso semaine dernière [%.0f %unit%]" { channel="linky:linky:local:weekly#lastWeek" } Number:Energy Linky_ConsoMoisEnCours "Conso mois en cours [%.0f %unit%]" { channel="linky:linky:local:monthly#thisMonth" } Number:Energy Linky_ConsoMoisDernier "Conso mois dernier [%.0f %unit%]" { channel="linky:linky:local:monthly#lastMonth" } Number:Energy Linky_ConsoAnneeEnCours "Conso année en cours [%.0f %unit%]" { channel="linky:linky:local:yearly#thisYear" } Number:Energy Linky_ConsoAnneeDerniere "Conso année dernière [%.0f %unit%]" { channel="linky:linky:local:yearly#lastYear" }

lolodomo commented 4 years ago

Please enable DEBUG logs, restart the binding and show your logs.

clinique commented 4 years ago

I had the same this morning. Linky thing went offline, stating "Communication Error", with a timeout error status. I suppose this can come from delayed or absent answer from the Enedis website, but this mean I'll have to strengthen the communication part.

Ozneb commented 4 years ago

I think the Enedis URLs have changed. "https://espace-client-connexion.enedis.fr/auth/UI/Login" is unreachable. May you update the addon with the good URLs ? or had a parameter to update them from config file or configuration screen ? Thanks

clinique commented 4 years ago

I have the same behavior than you in browser but debugging of the binding succeeds on the login page :-/

romainreignier commented 4 years ago

Enedis recently disabled this connection method. The recommended API is now https://datahub-enedis.fr/data-connect/ See these similar issues on other related projects:

clinique commented 4 years ago

Ok, thanks for the informations. I'll check this as soon as possible.

clinique commented 4 years ago

I'm afraid it will have to stay like this. Apparently accounts created before a given date still work for the binding. New accounts are switched to the new web service, that is behind a captcha, so I do not know how to bypass it. On the other hand, Enedis-Hub does not seem to be designed for our purpose.

lolodomo commented 4 years ago

Ok, it explains why it is still working for us. We can probably not expect a long lifetime for this binding unfortunately...

Ozneb commented 4 years ago

I don’t understand why it’s not possible to migrate to datahub. The documentation explain how the connection work with the new API and how to import all the user’s data with the Data API. Yes it’s a lot of work, but if you need help, just ask .

romainreignier commented 4 years ago

From what I understand, to use the new API, out of the sandbox, you need to be an enterprise an sign a contract with Enedis.

17 mai 2020 13:16:10 Ozneb notifications@github.com:

I don’t understand why it’s not possible to migrate to datahub. The documentation explain how the connection work with the new API and how to import all the user’s data with the Data API. Yes it’s a lot of work, but if you need help, just ask . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[https://github.com/openhab/openhab-addons/issues/7610#issuecomment-629780740], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ACDPSYCWLQYLOYGXQS4DBYTRR7BPVANCNFSM4M6ZJXEA]. [https://github.com/notifications/beacon/ACDPSYA5XW2XG2B2EKHV2PDRR7BPVA5CNFSM4M6ZJXEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEWE3CBA.gif]

Ozneb commented 4 years ago

Here my discussion with Enedis Support Team (in French so use Google Trad for English version ;) )

My Question : "Bonjour, Je souhaiterais accéder à mes données (relevés, historiques, …) via vos API afin de les intégrer dans mes outils domotiques. Comment puis-je procéder ?"

Enedis Answer : " Bonjour, Malheureusement, pour l’instant, Data Connect est réservé aux personnes morales. Nous n’avons pas encore adapté notre modèle de contrat ni automatisé les processus de gestion d’application pour permettre une utilisation pour des clients particuliers. Nous étudions ce point pour identifier un moyen efficace qui permettra cela de façon automatisée et sécurisée. Si vous le souhaitez, je peux noter de revenir vers vous quand nous aurons avancé ? Bien à vous, Fabien, de l’équipe Data Connect"

So, no personnal solution for the moment.

Ozneb commented 4 years ago

The long version of the Enedis answer : "Bonjour,

Je comprends votre besoin et votre déception de ne pas disposer d’API autour de vos données personnelles.

Nous avons le sentiment d’avoir beaucoup avancé, avec l’ERL et les spécifications de la TIC publiquement disponible depuis 2015, l’ouverture de l’espace personnel sécurisé en 2016, l’ouverture du portail SGE et de ses webservices à toute tierce partie en 2017 puis l’ouverture d’API type OAuth aux tiers en 2019. En particulier, par rapport à nos homologues européens ou internationaux, nous sommes parmi les rares à proposer autant de solutions ouvertes à tous. Des box domotiques ont déjà intégré nos API pour proposer à leurs utilisateurs d’accéder à leurs données personnelles, à côté de start-ups, collectivités, associations ou autres entreprises.

Néanmoins, vous avez raison et nous ne devons pas nous reposer sur cette avance par rapport aux autres utilities et continuer à avancer sur ce qui manque (notamment en se comparant aux acteurs du numérique plutôt que ceux de l’énergie). Nous avions bien identifié le besoin d’ouverture de nos API aux particuliers capables de développer eux-mêmes des interfaces ou de les intégrer eux-mêmes dans des outils existants. Cela nécessite un changement d’échelle car au lieu d’avoir un client « entreprise » qui développe un service pour plusieurs centaines ou milliers de clients résidentiels, cela nécessite de pouvoir gérer directement les clients finaux.

En toute transparence, les enjeux que nous avons identifié pour réussir cela sont :

Cela fait bien partie des fonctionnalités que nous souhaitons mettre en œuvre, mais après avoir sécurisé tous les points précédents.

Je note en tout cas votre intérêt et retiens de vous prévenir quand nous serons prêts à avancer opérationnellement sur ces points. Nous pouvons bien sûr échanger de vive voix, n’hésitez pas à me dire si vous le souhaitez."

clinique commented 4 years ago

Ouaip, ben c'est pas bon signe...voilà un binding mort né.

romainreignier commented 4 years ago

Merci @Ozneb C'est sympa de la part d'Enedis d'avoir fait une réponse complète mais effectivement ça ne sent pas bon...

lolodomo commented 4 years ago

@clinique : can you have a new look, it looks like it does work anymore for us too :-( Login is redirected.

clinique commented 4 years ago

@lolodomo : yes, I discovered this a few days ago also. Will try to take a look, but I wonder if this is worth the effort in the current state of the API. The audience will be very limited (maybe only you and me :) )

ParisYank commented 4 years ago

No doubt there are several of us out here who would like to have a download of our own data from Enedis.

Tukks commented 3 years ago

@clinique Now it connect well, there is no captcha on the website. image but the items stay UNDEF image

The GET request now is https://apps.lincs.enedis.fr/mes-mesures/api/private/v1/personnes/MKY506ZLW/prms/25109261799450/donnees-energie?dateDebut=17-10-2017&dateFin=1-10-2020&mesuretypecode=CONS

With cookie authentification, the response is ALL the data in a big json

ps : I have a new account created in june

clinique commented 3 years ago

Terrific ! Will take a look at this ASAP

clinique commented 3 years ago

@Tukks

The GET request now is https://apps.lincs.enedis.fr/mes-mesures/api/private/v1/personnes/MKY506ZLW/prms/25109261799450/donnees-energie?dateDebut=17-10-2017&dateFin=1-10-2020&mesuretypecode=CONS

With cookie authentification, the response is ALL the data in a big json

I do not get the same, if I try this url, I land here and there is still a captcha : image

Tukks commented 3 years ago

Weird, on a private window, I have the captcha. On a non private window I don't have the captcha :

image

I think one of my extensions is blocking the captcha url (maybe ublock??)

Tukks commented 3 years ago

Blocking the captcha does not work too, I don't understand why they show the captcha or not : image

Ok i tried something, and it work : 1) go to https://apps.lincs.enedis.fr/mes-mesures/api/private/v1/personnes/MKY506ZLW/prms/25109261799450/donnees-energie?dateDebut=17-10-2017&dateFin=1-10-2020&mesuretypecode=CONS connect and resolve the captcha 2) go to https://mon-compte-particulier.enedis.fr/home-connectee/ and click on "me deconnecter" 3) go to https://apps.lincs.enedis.fr/mes-mesures/api/private/v1/personnes/MKY506ZLW/prms/25109261799450/donnees-energie?dateDebut=17-10-2017&dateFin=1-10-2020&mesuretypecode=CONS and there is no captcha

How much times there will be no captcha, I don't know

clinique commented 3 years ago

Happily, I found this and successfully transposed it to Java, so Linky binding will soon be back.

lolodomo commented 3 years ago

@clinique : can you please confirm that the revival of the binding is only available in OH3?

clinique commented 3 years ago

Yes, this PR is only targetted to OH3. If somebody's willing to port it for 2.5.10, fine for me.

lolodomo commented 3 years ago

@clinique : following your instructions, when I go back, I see the expected page a very short time and then quickly I am redirected to https://apps.lincs.enedis.fr/saml/SSO, a page containing "État HTTP 405 ? Méthode non autorisée". In this page, I don't find the expected cookie and no entry "internalAuthId".

clinique commented 3 years ago

@lolodomo : funny enough, I faced the same issue today. I did install the Linky binding and followed my insctructions. Apparently the behaviour of their website did change in between. I then discovered that when you arrive on the webpage presenting your username and password, the cookie with internalAuthId is already present (I think it is generated after the captcha page). Can you confirm so we update the document ?

lolodomo commented 3 years ago

I just tried retrieving internalAuthId from the cookie in the login page and it seems to work. Fantastic.

Note that I have not always the captcha page, I got it just once I believe.

At server startup, I got now these warnings, they are probably relative to the Linky binding as I did not have them before, I believe:

22:42:48.929 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.948 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.968 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.972 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.975 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.993 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_DSS_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.997 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.009 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_DSS_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.012 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.040 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.060 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.072 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.075 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_GCM_SHA384 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.080 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_GCM_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.084 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.088 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.091 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.125 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.174 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.177 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]

Does it make sense for you ?

lolodomo commented 3 years ago

If I restart the server without the linky thing, these warnings are not appearing. So there is a clear link with the binding.

lolodomo commented 3 years ago

Addtionally, I got this warning:

22:53:13.521 [WARN ] [.core.thing.internal.ThingManagerImpl] - Initializing handler for thing 'linky:linky:local' takes more than 5000ms.

There is certainly a problem in the initialize() method of the thing handler, something taking too much time, maybe something which is no more scheduled in a separate thread ?

clinique commented 3 years ago

I just tried retrieving internalAuthId from the cookie in the login page and it seems to work. Fantastic.

Note that I have not always the captcha page, I got it just once I believe.

At server startup, I got now these warnings, they are probably relative to the Linky binding as I did not have them before, I believe:

22:42:48.929 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.948 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.968 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.972 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.975 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.993 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_DSS_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:48.997 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.009 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_DHE_DSS_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.012 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.040 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.060 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.072 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_ECDH_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.075 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_GCM_SHA384 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.080 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_GCM_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.084 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.088 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA256 enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.091 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.125 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_256_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.174 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]
22:42:49.177 [WARN ] [tty.util.ssl.SslContextFactory.config] - Weak cipher suite TLS_RSA_WITH_AES_128_CBC_SHA enabled for SslContextFactory@36524f[provider=null,keyStore=null,trustStore=null]

Does it make sense for you ?

@lolodomo : in the browser, the captcha is displayed only the first time until the cookie is generated or you clean your browser cache.

Yes, I know these warnings, they are generated from this line : 

httpClient.getSslContextFactory().setExcludeCipherSuites(new String[0]);

It was the only way I found to make jetty http client work with Enedis API (do you remember the attempts you made to remove okhttp and okio ?).

lolodomo commented 3 years ago

Yes, I know these warnings, they are generated from this line :

httpClient.getSslContextFactory().setExcludeCipherSuites(new String[0]);

It was the only way I found to make jetty http client work with Enedis API (do you remember the attempts you made to remove okhttp and okio ?).

Oh yes, I remember, a lot of time for no result ... and you found a simple solution with only one line of code :) Ok for these warnings about weak cipher.

Can we at least fix the init to avoid something taking too much time in initialize() ?

clinique commented 3 years ago

Yes, I know these warnings, they are generated from this line :

httpClient.getSslContextFactory().setExcludeCipherSuites(new String[0]);

It was the only way I found to make jetty http client work with Enedis API (do you remember the attempts you made to remove okhttp and okio ?).

Oh yes, I remember, a lot of time for no result ... and you found a simple solution with only one line of code :) Ok for these warnings about weak cipher.

Yes, just a simple line but so many searches to arrive here. I was kind of happy not to have to revert back to okio and okhttp :)

Can we at least fix the init to avoid something taking too much time in initialize() ?

Yes, I'll post a PR with documentation modification and initialize enhancement.

wborn commented 3 years ago

It was the only way I found to make jetty http client work with Enedis API (do you remember the attempts you made to remove okhttp and okio ?).

A better way might be to mail them they are using weak ciphers that may compromise the security of all their users. :wink:

lolodomo commented 3 years ago

@clinique : I believe there is a problem in your new binding architecture. You have one unique HTTP client but each thing handler is starting and stopping it. In case you have only one thing (probably the case for 99℅ of users), no problem. In case you define several things, I doubt it will work well.

lolodomo commented 3 years ago

This morning, I am looking at the values and I see that my consumption for yesterday is UNDEF, same for consumption this week. Other items are set. Do you see the same thing ? So maybe a bug in your code for Sunday ?

clinique commented 3 years ago

Took a look at it. This is true. What I see is yesterday measurement is not always available (I have a NaN on the 25/10). I will have to take that in account.

clinique commented 3 years ago

I will push a PR with the elements and corrections mentioned here.