probot-home-assistant[bot]
commented
3 years ago smart_meter_texas documentation smart_meter_texas source (message by IssueLinks)
Closed NateEaton closed 1 year ago
probot-home-assistant[bot]
commented
3 years ago smart_meter_texas documentation smart_meter_texas source (message by IssueLinks)
probot-home-assistant[bot]
commented
3 years ago Hey there @grahamwetzler, mind taking a look at this issue as it has been labeled with an integration (smart_meter_texas) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
atkjedi
commented
3 years ago I also had this same issue , I also tried to remove and re-add and then after that could not re-create, would get invalid credentials.
logger: homeassistant.config_entries
Source: components/smart_meter_texas/__init__.py:49
First occurred: 8:11:40 AM (1 occurrences)
Last logged: 8:11:40 AM
Error setting up entry {account name} for smart_meter_texas
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
await waiter
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 528, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
self._sslobj.do_handshake()
File "/usr/local/lib/python3.9/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'smartmetertexas.com'. (_ssl.c:1129)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 304, in async_setup
result = await component.async_setup_entry(hass, self) # type: ignore
File "/usr/src/homeassistant/homeassistant/components/smart_meter_texas/__init__.py", line 49, in async_setup_entry
await smart_meter_texas_data.client.authenticate()
File "/usr/local/lib/python3.9/site-packages/smart_meter_texas/__init__.py", line 196, in authenticate
await self._init_websession()
File "/usr/local/lib/python3.9/site-packages/smart_meter_texas/__init__.py", line 147, in _init_websession
await self.websession.get(
File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection
raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host smartmetertexas.com:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'smartmetertexas.com'. (_ssl.c:1129)")]
johnavich
commented
3 years ago This issue was caused by SmartMeterTexas.com SSL Renewal process. They didn't include the SAN for "smartmetertexas.com", and only included "www.smartmetertexas.com". I have already emailed SMT support about their SSL issue(s), and hope to get a reply soon.
johnavich
commented
3 years ago Called their help line, they escalated the ticket to their L2 team, who deals with API and such. I was given an ETA of 48h for a fix, so we should just buckle down.
On that note, i've been looking through the code, and cannot locate where it looks for the domain to connect to. If someone could point that out, i'd be grateful.
grahamwetzler
commented
3 years ago That's surprising they're being this helpful. We're using the unofficial API used by the web browser. Thanks @johnavich.
The API logic is performed by the library I created. https://github.com/grahamwetzler/smart-meter-texas
Indianb0y016
commented
3 years ago Was crawling through the issue list just to find something related. Glad others are using this integration. SMT really needs to get an official api up and running cause the whole point of it is to be smart meter monitoring.
grahamwetzler
commented
3 years ago Well, they do have an official API but it has some ridiculous requirements that would make it much more difficult to offer an HA integration.
github-infohost-org
commented
3 years ago When I put https://smartmetertexas.com into a web browser, it redirects to https://www.smartmetertexas.com This redirect would have been put in place by Smart Meter Texas network/internet engineers for a reason.
Since this seems to be the way they want their website to be reached...
Maybe in the CONST.PY change the following: BASE_HOSTNAME = "smartmetertexas.com" to BASE_HOSTNAME = "www.smartmetertexas.com" to fix this issue?
Yes...No...???
NateEaton
commented
3 years ago The challenge is that it took removing www from BASE_HOSTNAME to get this integration working again earlier this year. See Smart Meter Texas Integration Failed to Connect #40348. Not that something didn't change again on the SMT side... perhaps things reverted to an earlier state on their end.
kitprice
commented
3 years ago The challenge is that it took removing www from BASE_HOSTNAME to get this integration working again earlier this year. See Smart Meter Texas Integration Failed to Connect #40348. Not that something didn't change again on the SMT side... perhaps things reverted to an earlier state on their end.
This is still an issue, so it is a no-go adding the www back to BASE_HOSTNAME. I verified this by modifying the library and testing.
I also verified that by disabling strict checking (ssl_context.check_hostname = False) in the library, the test commands succeed. So, if SMT gets the SSL certificate SAN updated correctly, everything should get back to normal.
Edit: Corrected "removing the www from BASE_HOSTNAME" to "adding the www back to BASE_HOSTNAME".
meyerrj
commented
3 years ago When I put https://smartmetertexas.com into a web browser, it redirects to https://www.smartmetertexas.com This redirect would have been put in place by Smart Meter Texas network/internet engineers for a reason.
Since this seems to be the way they want their website to be reached...
Maybe in the CONST.PY change the following: BASE_HOSTNAME = "smartmetertexas.com" to BASE_HOSTNAME = "www.smartmetertexas.com" to fix this issue?
Yes...No...???
Unfortunately this does not work for whatever reason. Part of the fix to get SMT working recently was to actually remove the www prefix on the domain. We also had to add in additional logic to work around validations with their existing SSL certificate on smartmetertexas.com
My guess is there is some server-side validations or filtering on the www subdomain that prevents it from working with the library.
Edit: www.smartmetertexas.com appears to be behind Akamai Edge, while smartmetertexas.com appears to go directly to the host. It's likely something with the Akamai application security controls getting triggered causing the block/interference. https://myakamai.force.com/customers/s/article/Why-is-Akamai-blocking-me?language=en_US
johnavich
commented
3 years ago So i reached back out to SMT, and they mentioned they're getting quite a few calls regarding the SSL certificate. So we're not the only ones affected. I did recommend they lower the API requirements however, as that would make things like this much easier to navigate, and possibly provide us with more information.
I'm hoping that we can get scraping data to allow for surplus generation data to be available on HA, but for now, that's a pipe-dream.
livetoautomate
commented
3 years ago I am also having this issue. Did SMT support give any indication if they will update the SSL certificate with the SAN?
johnavich
commented
3 years ago So, long story short, SMT is being billeigerent, trying to continue requiring a publicly signed cert and a static ip address.
Until they come to their senses, I did figure out a workaround that worked for me. I'll be pasing that to the "fix" on Graham's original site.
johnavich
commented
3 years ago 
i was able to get data from SMT using my "hack". this did require me updating the internal SMT code from graham, so YMMV.
ricardouseche
commented
3 years ago @johnavich Would you mind sharing what the fix looks like? At this point, SMT is super unreliable and I'm open to any alternatives.
johnavich
commented
3 years ago the hack was to change the hostname endpoint back to www. and update my DNS to send www to the IP for non-www. i call it a hack because if i update my CORE again, it'll break my fix, and i'll have to re-do it until graham's code is updated on the backend.
meyerrj
commented
3 years ago I believe the proper fix to this would be to get the code to play nice with Akamai's filtering. As far as I can tell, SMT is essentially using Akamai as a reverse proxy to their non-www hostname.
If SMT ever decided to restrict http traffic to Akamai's proxied requests, we'd be dead in the water again. The hostname/DNS hacks are really a short term solution at best, and aren't really appropriate as a fix to be deployed with HA.
johnavich
commented
3 years ago be that as it may, i'm not skilled enough to work through whatever akamai's issues are, and SMT support is unwilling to work with us to determine why its broken. If needed, we could start filing PUCT complaints about this, and maybe get a fix, but we won't get anything directly from them.
musabbari
commented
3 years ago @johnavich If it is something easy to explain, can you please let us know which file/line did you change the hostname endpoint back to www., and how do you change DNS table on Home Assistant?
johnavich
commented
3 years ago @musabbari I had to get the working directory for the homeassistant docker container "docker inspect homeassistant | grep MergedDir", then in that directory, modified usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py and added www. to the BASE_HOSTNAME variable.
As @meyerrj has the right of it, that this is a short-term temporary fix that has the potential to break at ANY time, i would use caution.
monkmad
commented
3 years ago @johnavich Thanks for the detailed instructions, works like a charm 🙂
@meyerrj, a request/suggestion for you: The proper fix would be for the SMT folks to come to their senses and manage their SSL certs correctly. Since we don't have control over that, I'm wondering if we can incorporate the fix suggested by @johnavich with a config option, which toggles it off or on, depending on the state of the URL/IPs? This would help many other users, who may not be comfortable with changing code themselves.
johnavich
commented
3 years ago I highly doubt meyerrj will devote any resources to make that kind of change. And i don't blame him. He's correct that its entirely possible that SMT will get their firewall in order and block non-akamai traffic, which would leave us all in the same boat. I simply made this change to my system, as i accept the risks inherent in my setup. If you are not comfortable making these changes at that level, i don't recommend having these "changes" available to the end user at all.
meyerrj
commented
3 years ago @monkmad While it might be possible to make that change in the HA implementation, it involves changes to the supporting library as well. I'm also not sure what the HA policies are on such configuration options, but I imagine as long as it's not the default it would be fine. The unit testing would be a PITA because it would fail by default as well.
@grahamwetzler is the owner of the supporting library, so he would need to weigh in on this, as I am merely a contributor that helped get it working again before the certificate expired and was replaced.
I did try taking a crack at getting the library code to work with Akamai, but it is not a simple problem to solve.
monkmad
commented
3 years ago Yep, I also tried some cursory research into the Akamai Edge CDN setup and either bypassing it or setting the JS cookies. Apparently, that's not easily possible unless you are an Akamai customer. I saw some Python libraries and some header options which probably could be used, but without knowing what values they expect, it's a needle in a haystack. The other option is to use a headless browser to render the JavaScript and set the cookies, but that's probably overkill for what's supposed to be a simple addon.
monkmad
commented
3 years ago Since making the change suggested by @johnavich, I've updated my HA install a few times and of course, lost the manual changes each time. so, I decided to semi-automate the process of finding the const.py file and updating it. Here is the result, one command to re-apply the changes after updating to a new version. Enjoy 😉 (I'm sure the command can be optimized, and might have some unforeseen bugs too. Please post your improvements)
Disclaimer: Use at your own risk, I'm not responsible for anything, including the Zombie Apocalypse...
sed -i 's/"smartmetertexas.com"/"www.smartmetertexas.com"/' $(docker inspect homeassistant | grep MergedDir | sed 's~"MergedDir": ~~g; s~\"~~g; s~,~~g; s~ ~~g; s~$~/usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py~g')
Does anyone know how to create a "post update" script in HA to automate the above command after each update?
kckopp
commented
3 years ago Does anyone know how to create a "post update" script in HA to automate the above command after each update?
For this I used a platform start automation that checks to see if the changes exist and makes them if not. This is based on what I found here. Detail below.
I'm not sure that I needed to loop the hosts update the way that I did but in my first few tries it seemed as if during startup my hosts addition was being stomped by the system. Update: I've confirmed that the loop was not needed and that what I was observing was that the platform start automation runs later than I thought that it would, only once "Home Assistant has started!" message appears. SMT will hang up the startup for a few minutes before it will move to retrying in background, allowing startup to finish, at which point SMT will load as the hosts file has been updated. I've removed the loop from the hosts.sh below.
Also, in testing I found that If I removed www from the BASE_HOSTNAME variable the platform start automation would add it back but not before the component had already loaded and failed due to the SSL error. A second restart resolves the issue.
As all have pointed out, this is a temp fix, could break at anytime, use at your own risk, zombies, ...
automation:
- alias: Update hostname and hosts for Smart Meter Texas
trigger:
- platform: homeassistant
event: start
action:
- service: shell_command.smt_host
- service: shell_command.hosts
mode: single
shell_command:
hosts: bash /config/hosts.sh
smt_host: bash /config/smt_host.sh/config/hosts.sh
#!/bin/bash
if ! grep -q 12.153.148.59 '/etc/hosts';
then
echo '12.153.148.59 www.smartmetertexas.com' >> /etc/hosts
fi/config/smt_host.sh
#!/bin/bash
if ! grep -q www '/usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py';
then
sed -i 's/"smartmetertexas.com"/"www.smartmetertexas.com"/' '/usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py'
fi
WWE-Corey
commented
3 years ago @kckopp does this work with OS version? I'm not finding the specified directories.
johnavich
commented
3 years ago @WWE-Corey That implementation is specific to HomeAssistant.
kckopp
commented
3 years ago does this work with OS version? I'm not finding the specified directories.
@WWE-Corey It works in HassOS. In HassOS if you want to see paths mentioned in the shell scripts, they are in the homeassistant container.
docker exec -ti homeassistant /bin/bash
WWE-Corey
commented
3 years ago Thanks. Guess it won't work for me then. I have Home Assistant OS 6.6 installed in a VM. tried the command for giggle but came up an error since I do not have docker installed. I can't even seem to find anything for the integration.. was going to try to manually edit and see if I could connect again.
kckopp
commented
3 years ago Thanks. Guess it won't work for me then. I have Home Assistant OS 6.6 installed in a VM. tried the command for giggle but came up an error since I do not have docker installed. I can't even seem to find anything for the integration.. was going to try to manually edit and see if I could connect again.
I'm running Home Assistant OS 6.6 on bare metal (NUC). To run the docker exec command I'm using the SSH & Web Terminal add-on though I'm sure there are other methods you could use to get a root shell in your Home Assistant OS host VM. Home Assistant OS runs Core in a Docker container.
johnavich
commented
3 years ago @WWE-Corey you can access the raw console and hit Alt+F2 to get access to the root shell. From there you can get to the appropriate docker implementation.
I run mine through proxmox KVM and was able to do this.
WWE-Corey
commented
3 years ago I have not figured it out yet but still trying. I was able to go through the proxmox console, hit alt-f2 and get to the root but did not find the /usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py or any python related directories for that matter. Maybe I am over thinking this.
kckopp
commented
3 years ago I have not figured it out yet but still trying. I was able to go through the proxmox console, hit alt-f2 and get to the root but did not find the /usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py or any python related directories for that matter. Maybe I am over thinking this.
Once you have a root shell in the Home Assistant OS Host you can find the files you are looking for in the Home Assistant Core container.
johnavich wrote:
I had to get the working directory for the homeassistant docker container
docker inspect homeassistant | grep MergedDir, then in that directory, modified usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py and added www. to the BASE_HOSTNAME variable.
Or docker exec -ti homeassistant /bin/bash to get a shell in the container.
WWE-Corey
commented
3 years ago Sheesh, sorry for that and thank you for the help. I'm not a docker person.. tried for a while but pretty much abandoned a while back.
I have it all added and restarted and then rebooted the VM too. No errors on the shell scripts but the smartmeter integration shows not-loaded and not seeing any new data (yet) in the Energy tab. Is this normal or is there something I should could check?
edit: Seems it loaded the entity after a few minutes and got the total usage. Just waiting for hourly updates now - Thank you again!
github-actions[bot]
commented
2 years ago There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
NateEaton
commented
2 years ago The root issue as I understand it (mismatch between URL and cert) still seems to be an issue.
Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection transport, protocol = await self._create_connection_transport( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport await waiter File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 528, in data_received ssldata, appdata = self._sslpipe.feed_ssldata(data) File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata self._sslobj.do_handshake() File "/usr/local/lib/python3.9/ssl.py", line 944, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'smartmetertexas.com'. (_ssl.c:1129)
johnavich
commented
2 years ago This is not going to be fixed, as its a SmartMeterTexas issue, and security should not be compromised for the sake of a very few of us. Some of us are working on how to get around the WAF hosted by Akami, but for now, we have a backup option that is working for us. Please see my earlier comments to determine how/why to use this backup option.
This issue should be closed as Won't Do.
somesmart
commented
2 years ago Does the workaround still work for you @johnavich ? I am trying to add the integration and I don't get the error any longer but it just spins endlessly.
johnavich
commented
2 years ago @somesmart i hadn't upgraded to the most recent 2022.03.x release, and i was still getting data from SMT. So if your experiencing an issue, make sure you have updated the const file, then restarted the homeassistant docker image. I find it will not work after updating the const.py file before restarting the container. If something changes (after i've upgraded to 2022.3.3), I will update, and attempt work through it.
mwjones1971
commented
2 years ago @somesmart I have it the integration working with 2022.3.3 using the patch automation, although it still requires a second restart after each upgrade for it to work again (since the patch automation is applied when HA restarts).
@johnavich I don't know if you've noticed this, but when restarting HA, it takes a long time for the integration to start. This doesn't seem to be a new thing as its been happening across several releases now (I'm using a HA Blue appliance).
Likewise, the integration has been picky, many times missing intervals, and then I get hit with an interval that catches up on all the missed intervals, creating a spike. Once I get a house purchased and out of this rental, I already have bought a Sense monitor to do my energy usage monitoring, and will go away from SMT (of course where I'm looking to buy is also not serviced by one of the SMT partner companies so they wouldn't work anyway).
somesmart
commented
2 years ago @mwjones1971 @johnavich Thanks for the replies. I've confirmed the change in the const.py and restarted the container but I'm still getting the same result. I'll keep playing with it but it's good to know it should be working if I can figure out my issues!
Edit: I had an error in my configuration.yaml and it's working for me now. Thanks again!
johnavich
commented
2 years ago @mwjones1971 I get that too, i've yet to be able to explain it. However the general code for HA has a 30 minute limiter, so it only checks once every 30 minutes. If a check-in fails for whatever reason, it's a full hour before the value is updated. Plus, during a multi-hour missing dataset, i believe they're doing maintenance, so we WONT get that data, not until the maintenance is resolved.
kckopp
commented
2 years ago I've updated my patch automation since I would often upgrade HA, get distracted, forget to restart a second time and then miss out on a bunch of SMT data. I now update a binary sensor when, during upgrades, the const.py is updated by smt_host.sh. In a new automation, I then trigger the second restart based off of this binary sensor. I'm hoping others find this useful.
automation:
- alias: Update hostname and hosts for Smart Meter Texas
trigger:
- platform: homeassistant
event: start
action:
- service: shell_command.smt_host
- service: shell_command.hosts
mode: single
- alias: Restart on SMT Host Update
trigger:
- platform: state
entity_id: binary_sensor.smt_host_updated
to: 'on'
for: "00:00:30"
action:
- service: homeassistant.restart
mode: single
shell_command:
hosts: bash /config/hosts.sh
smt_host: bash /config/smt_host.sh/config/hosts.sh
#!/bin/bash
if ! grep -q 12.153.148.59 '/etc/hosts';
then
echo '12.153.148.59 www.smartmetertexas.com' >> /etc/hosts
fi/config/smt_host.sh
#!/bin/bash
if ! grep -q www '/usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py';
then
sed -i 's/"smartmetertexas.com"/"www.smartmetertexas.com"/' '/usr/local/lib/python3.9/site-packages/smart_meter_texas/const.py'
curl -X POST -H "Authorization: Bearer LongLivedAccessTokenFromProfile" -H "Content-Type: application/json" -d '{"state": "on", "attributes": {"friendly_name": "SMT Host Updated"}}' http://127.0.0.1:8123/api/states/binary_sensor.smt_host_updated
fiedit: while it worked out in testing and appears to have fired after an actual update the restart didn't happen, I've added that the trigger must be on for 30 seconds hoping that solves the issue
update: I've just updated to 2022.3.6 and the second restart happened, it looks like waiting a bit may have solved this
rfnovo
commented
2 years ago Got an easier workaround for all that. Just change the "check_hostname" from True to False and keep the "BASE_HOSTNAME" as "smartmetertexas.com". To change those values without having to mess with the HASS container I just created a Custom Integration combining the Core integration and the API with the adjusted parameters; https://github.com/rfnovo/Smart_Meter_Texas
johnavich
commented
2 years ago the reason why this idea has been rejected is because it opens users up to potential security issues. If users want to do this, then that is on them, which is why it won't be considered as a fix.
rfnovo
commented
2 years ago Agreed. Is just a workaround
neclimdul
commented
2 years ago maybe a dumb question, looks like @grahamwetzler might have addressed this in the latest release?
https://github.com/grahamwetzler/smart-meter-texas/blob/main/smart_meter_texas/const.py#L3
Does home assist just need to update?
The problem
Smart Meter Texas integration has been working (including with Energy Dashboard since 40348 was resolved but as of late yesterday, is now reporting Failed to Connect with an SSL Certificate Error.
What is version of Home Assistant Core has the issue?
core-2021.10.2 and core-2021.10.4
What was the last working version of Home Assistant Core?
core-2021.10.2
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Smart Meter Texas
Link to integration documentation on our website
https://www.home-assistant.io/integrations/smart_meter_texas/
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
This integration has been working fine since July and stopped working last night (2021.10.11 about 9pm CT), still not working this afternoon even after updating Core from 2021.10.2 to 2021.10.4. I am able to log into my account on smartmetertexas.com.
July
Yesterday