Smart Module ends communication after certificate is given

[Boohooligan]

Just for good measure, I wiped the raspberry pi 3b+ my previous attempts were set up on and did a fresh install of DietPi OS and Docker. Cloned your repo, built it, configured the docker-compose.yml. Here is where I'm at with it.

Pi-hole is successfully routing energysmartwaterheater.com and devices.irissmarthome.com to the RPi.

MQTT is showing online in home assistant and the energysmart topic is available, though only showing status=online.

Wireshark is showing the smart module contacting the server, initial handshake (syn, syn ack, etc), then a client hello over TLSv1, the nginx proxy responds and provides the certificate, then the smart module sends a fin ack, indicating it's done. It's never offering up any of the data.

Proxy container IS able to ping the bridge container. The bridge does fail several times to set up the listen on port 8001 before finally working, unsure if this is normal behavior or not.

Unsure how to get more detailed logging from the nginx proxy.

[starsoccer]

It sounds honestly like the device isnt liking the proxy cert or something along those lines. I would personally still triple check the nginx version its using.

Assuming that its the correct version still, next I would personally try making a request by hand to /~branecky/postAll.php with some data and seeing if the bridge picks it up and atleast shows it was hit

[Boohooligan]

So that was maybe fruitful. nginx is on 1.24.0 as the dockerfile specifies. I used Fiddler to post some test data and then checked the logs on the proxy. I wasn't seeing anything before on the logs BUT now I'm seeing this:

2024/01/25 13:04:11 [error] 8#8: *399 upstream timed out (110: Connection timed out) while proxying connection, client: 192.168.50.123, server: 0.0.0.0:443, upstream: "192.168.0.241:8001", bytes from/to client:141/0, bytes from/to upstream:0/0

This is the test request from my laptop. The thing is that I adjusted the docker-compose.yml to use energysmartbridge_energysmart-bridge_1:8001 as the proxy pass location so I'm not sure why it's using your IP address. Did you hard code that somewhere in the files?

Also, I am seeing requests from the smart module in the error list too now, which I wasn't before. I'm also not sure why the system time in the logs seems so off. The system time and timezone on the RPi are correct.

[starsoccer]

Yes I did change the Ips on my fork, long term Id like to make it customizable but currently I just have it hard coded which Isnt ideal. If you open the docker-compose.yml under enviorment you can change the IP there, and then in the nginx.conf.template there is the IP again which you can change.

I dont really know C# so I am kind of just brute forcing my way through this so apologies for breaking things along the way. It may be best if you take a day or two while I break things and get them into a slightly better state

[Boohooligan]

I changed the IP in the nginx.conf.template file and removed the resolver since I want it to just use the container's default resolver since that's how I tested the ping between containers. Did another fiddler test and the bridge did receive and log the data. I think something happened overnight while I left it running previously because the module isn't sending requests to energysmartwaterheater.com frequently anymore, just the heartbeat every minute or so to devices.irissmarthome.com. With it like this, I won't know if this works until it tries to use energysmartwaterheater.com to post data. I'll leave it running for the day and see if it posts at some point. Looking at my pi-hole logs it goes hours between post attempts sometimes. Before though, it was reliably every 5 minutes. I wonder if the time zone difference between the docker container and my local systems made the certificate not valid yet or something and it wasn't until overnight that it became valid.

I'm not a developer exactly, though I do know a number of languages and did learn some C# about 12 years ago now. I mostly do IT Operations but a lot of homelab type stuff. I'd love to help make this something people can easily just drop in place and have it work if I can. Once I have it working on my side I think it'll be a good opportunity to make changes and determine what works and what doesn't. These modules are so cheap on eBay it would be a shame for them to all just end up as more eWaste and the hardware hacking method the dude on instructables did was not appealing to me at all. A software solution like this is MUCH more elegant.

[starsoccer]

I am not sure how relevant the pi hole dns logs are to be totally honest. While its possible it looks it up every single time via the dns server(pi hole), Id assume at some point it wouldnt and iit would just cache the results locally but I could be totally wrong.

Usually it hits the device every ~5 minutes or so. If you thing its being finicky my suggestion would be to kill power to your water heater, and then give it a few seconds and then turn it back on and see if anything changes. Otherwise just hard resetting it and reconnecting it to wifi could work too.

[starsoccer]

Re improvements, at the moment I am working on trying to get the 6 items on my list from the other thread done. Specifically I am working on trying to get auto discovery working with HA. Right now basically a MQTT device appears but it has no entities and the entities all appear attached to no device.

Once I get that resolved I will try to improve the code a bit and clean it up as I am just making things a bit of a mess right now and then will work on making the IP configurable and then hopefully turning this into a HA addon

[Boohooligan]

It's working now! It auto created 9 entities as well in Home Assistant. Now that that's working, let me know what I can do to help.

[starsoccer]

Woohoo yes I just got the auto discovery to work so your the first person to have it work besides me 😁

Uhm as for what else you can help with, right now I am going to work on changing the nginx and docker compose to use an env variable instead of my hard coded ip. Once That is complete Id like to work on converting this into a home assistant addon. The complexity with converting it to an addon is that right now its 2 dockers which I think will likely need to be converted to one

[starsoccer]

Actually it seems like my last change for using an env variable might have worked. Can you try using PROXY_PASS_LOCATION as an ENV variable and see if that works to change the IP/location of the bridge?

[Boohooligan]

I'm away from my house but I can do this when I am back later on today and will let you know.

[starsoccer]

Ive already started to work on this, no gurantees Ill get it though, my first thought/attempt is to use Docker in Docker to basically just run the other containers inside. Its far from ideal but is the easiest option.

Longer term Id love to remake this whole solution and not use C# and use something like nodejs or rust and then running everything in one container should be a lot easier.

[Boohooligan]

Definitely a solid goal. Having the 2 containers for sure complicates the situation. It seems like the proxy part is pretty simple so building a single container that can do that and the MQTT shouldn't be too hard for someone who knows this stuff. Before taking this on I'd never used docker before so didn't understand how it all worked together.

I think if we can get all of the entities auto populating in HA I could write some really good step by step instructions for people to get it all working. I'm pretty good with process documentation.

Definitely better than trying to hack the controller or water heater directly. I'm sure there's many people who would love to do this with HA.

Actually, I wonder if a direct integration with HA would be possible. I'm not sure if anything uses port 443 on HA by default. People could do internal port forwarding on their router to use a different port, if the router supports it.

[starsoccer]

I think I was able to get it working. I just pushed a new change. You can ignore the docker compose for now and just use the top level dockerfile. This should build a single image with both in it. From my limited testing it seems to be working.

Next I am going to work on trying to get this working in HA as an addon.

I think most of the entities are already populating. Is there something else missing you want?

Regarding a direct integration, Id say probably not likely since this only talks to some obscure domain and requires dns configuration, etc.. I think the best case scenario right now is really to just get this as a HA addon and then anyone can simply install the addon, setup the DNS to redirect to the HA addon and then have things work

[Boohooligan]

Sorry, I meant something more like an addon and not an integration. Could be something we put in HACS.

The mode is incorrectly reporting as heat pump while it's actually in hybrid. It may be that my heater has modes that weren't accounted for but I won't know until I get home and can look at the raw data it's sending.

I'll try the one container solution when I get home and see if it works for me like that as well. It'll be a few hours.

I built an esp32 device to read the LEDs to tell me what mode it's in and a switchbot bot to press the button to change based on ambient garage temperature. If I can do all of that accurately through this I'll call it a win.

[starsoccer]

HACS from my experience tends to be more for custom integrations, because of how this works I dont think there is anyway for us to get away from needing an addon of some sort honestly so I think an addon is the easiest option.

Regarding the mode, so that is sort of accurate lol. I mean I can expose the actual mode if you want but basically home assistant has predefined modes for hot water heaters and you have to use them, you cant define your own, https://www.home-assistant.io/integrations/water_heater.mqtt/#modes - which I am currently mapping here, https://github.com/starsoccer/EnergySmartBridge/blob/master/EnergySmartBridge/Modules/MQTTModule.cs#L234 - Happy to revisit the mapping if you disagree with how I did them.

All good, hopefully by then I will have it working as an addon. Going to start on that shortly.

Hmm I am not sure why you would want to change the mode so frequently honestly. I would think its better to just change the temp you want and leave it in hybrid mode. Its worth noting that service supposedly lets you change some of the settings on the device like temp, but I have not been able to get any of those to work 🤷

[Boohooligan]

For the modes, my heater has efficiency (heat pump only), hybrid (heat pump plus electric coils), and electric. So it saying heat pump when it's in hybrid is confusing.

As for why I change it, I live in Florida and the heater is in my garage. When it's cold I don't want it to take forever to recover after a shower so I switch to electric. When it's in more of a middle range, like 70-75 I do hybrid, and then nearing 80 and up I do heat pump only. Most of the year it's in heat pump only but this time of year it's hot one day cold the next. It was to take the mental load off but still balance recovery time with efficiency.

[starsoccer]

I cant comment on what modes your unit has mine has 4 which it sounds like are similar or the same, Efficiency(Heat Pump Only),Hybrid(Both),Electric(Electric only),Vacation(off). Do you think it would be better to then have efficiency map to heat pump and then hybrid to eco? It just seemed odd to me.

Hmm interesting Ive never touched mine honestly but its in my basement though it can get cold down there, its unfinished.

[Boohooligan]

Ah, well mine also has vacation mode, I just didn't include that since it's not a normal operating mode. Since the options in HA don't match, I think Efficiency should map to heat pump and Hybrid to eco. That said, I'd make my own translation either way as I don't intend to really use the water heater "type" in HA. I would break out just the settings I want to monitor.

One thing I noticed is that I have a message in HA that the device doesn't have a unique ID so I can't change settings from the UI. That's probably fine either way since my settings would be changed through automation, but unclear if it'll work like that with the error. I haven't tested yet.

Just getting back home now. Going to stop my containers and try the one docker solution and will report back.

[Boohooligan]

So, I've cloned the repo as it is right now into a new folder but I'm not sure how to launch it with the new configuration. I edited the .ini file to have my MQTT settings but how do I run this?

This whole situation is my first experience with docker at all so I still know very little about it.

[Boohooligan]

Misunderstood. I installed it to my HA, there are a few issues so far.

The user/pass entered into the config file don't seem to actually get used to talk to the mqtt server or may be malformed. I got a slew of bad username or password errors on the broker while this was running.

MQTT_Server in the config I think should just be a string and not a url. I don't believe it needs to have http:// at the start. Also since it's going to be run inside HA it can probably be defaulted to 127.0.0.1.

[starsoccer]

Sorry about that I had stepped away for a bit. I just fixed the mqtt server not being a string and waiting on it to rebuild right now. Hopefully once thats done it will work as an HA addon.

As for how to use it now, sadly converting it to HA kind of breaks complicates the regular non HA usage. So if you want to use it without HA, your best bet is to just build the dockerfile under energysmartbridge yourself.

Its also worth noting you do not need to edit a .ini file to change any settings. Or atleast you shouldnt need to. You should be able to change any/all settings using HA or env variables.

Regarding the updated mapping, Ill swap that over once I confirm its working in HA

[Boohooligan]

Yup, I had misunderstood what the current state of the project was and thought I needed to build in the RPi.

Will update soon. Did you also fix the null username/password along with the url to string change? That's what was causing the login errors on the MQTT broker.

[starsoccer]

No I didnt fix the null username/password issue. I am not actually sure what is going haywire there trying to debug it now. So probably best to hold off until I post here saying its resolved.

On top of that issue, I also need to figure out if the addon is getting its own IP or not separate from HA. I am not sure about your HA setup, but mine is natively using 443. This container also needs 443 so I am not really sure how that is going to work exactly. I am hoping since ideally this addon will have its own static ip it wont be an issue though.

[Boohooligan]

Oh. I'm not sure what mine would be using 443 for except maybe connection to Nabu Casa? The default port is 8123.

[starsoccer]

Im not sure, it probably depends if you set it to use a different port, or if you have https enabled on it locally or not. In any case I am still messing with the addon trying to get it to work. Sadly the process is a bit slow as I have to keep tagging a new release, letting github rebuild the image, etc.. So it takes a good 10 minutes to cycle through a change

[Boohooligan]

All good! I appreciate you doing this. If this can work as an HA addon for people not using 443 already it's a coup. Not having to run a separate box for it is pretty huge.

[Boohooligan]

If this is working I'll do a video on it so other people can see it in action and I can talk through how to set it up, the caveats, etc, unless you'd rather do it.

[starsoccer]

Well 100% something is going to need to run on 443. It may end up being nginx proxy manager, or something else.

My hope is that with HA I can set this addon to run in bridge mode where it gets a IP from the host network so HA and this addon will have 2 different Ips and then we simply set dns for the addon IP and call it a day

[starsoccer]

If this is working I'll do a video on it so other people can see it in action and I can talk through how to set it up, the caveats, etc, unless you'd rather do it.

Sure feel free. My goal at this point is to get it working as an HA addon so I dont need to deal with this being on another machine running

[Boohooligan]

Well 100% something is going to need to run on 443. It may end up being nginx proxy manager, or something else.

My hope is that with HA I can set this addon to run in bridge mode where it gets a IP from the host network so HA and this addon will have 2 different Ips and then we simply set dns for the addon IP and call it a day

Ohh that's slick, so a virtual interface on the network. I like it.

[starsoccer]

So actually I realized I was mistaken my HA instance isnt on 443. Not sure why I thought it was. In any case I wasnt able to get the addon its own IP but it seems to work just using the same IP.

Can you try this latest version and let me know if it works for you? It seems to be working for me without any issues. I am going to probably try to clean up some code now and then make the mapping adjustment you asked for

[Boohooligan]

Heck yes! You are the MAN! Worked beautifully. I like the option on the config to change the port to use inbound for the proxy. If you want, I can help with some of the verbiage and instruction on the config page. I definitely would like when you have this in a 'ready' state like cleaning up code and stuff for the documentation to be really good for how to use it, potential issues, etc. I'm happy to write up that stuff.

I'm not super familiar with how to use github as far as collaborating but I can see if I do some change requests or something on it to help out with that stuff.

[Boohooligan]

So, another question I just thought of I guess. On mine, it reports remote administration is off. My water heater display also switches between the temp and "rA" (remote administration, I'm assuming). I wonder if the reason the settings won't change when called is that remote administration is not enabled. There may be some button combination on the water heater itself to change that, similar to how to change the '48 hour' thing where it resets back to hybrid automatically by default.

I suppose it could also just be that the method this uses isn't capable of writing back.

[starsoccer]

Yeah so changing the port I don't think works currently but I can try to tweak some things to get it too if you need it.

Re documentation, etc.. sure feel free to write some stuff up and Ill update the readme. Are you familiar with git at all?

Regarding remote administration, I wonder if my code has the values reversed. Can you try changing the temp or something in HA and see if it reflects in the unit? And see what the logs show?

[Boohooligan]

I've tried several times to change the temperature and mode using the home assistant water heater service with no change. I'm not sure what logs I'd collect for that. If you can give me more detail on where or how I'd collect them I'm happy to.

As far as familiarity with git, I've never really used it. I mostly just have the account because some HA addon or integration required it.

As for changing the port thing, I don't really need it, but could see a case where someone might. But also you could just say just use 443 or don't use it at all.

[Boohooligan]

I stand corrected, the water heater actually did change temperature, it was pretty slow to do it though. I will try to change modes again and see if that gets me anything.

[starsoccer]

Okay so the temp changed worked while remote administration was turned off? Can you try turning it on and see if.it still works? Maybe I have KT backwards then.

Regarding git, okay no problem so then maybe just make an issue for each for now and then I can update the repo.

As for the port, thinking about it more, it cant be changed because it has to be 443, so Ill make another change to remove it

[Boohooligan]

I'll mess with it some more and see what combinations of actions seem to work and not work.

For the port, it COULD work if someone has a router that can do internal port forwarding. They could do some iptables or something to reroute from the smart module on 443 to another port on HA. That said, the number of people who are going to do that and would need to do that is probably very small, if any.

[starsoccer]

I'll mess with it some more and see what combinations of actions seem to work and not work.

For the port, it COULD work if someone has a router that can do internal port forwarding. They could do some iptables or something to reroute from the smart module on 443 to another port on HA. That said, the number of people who are going to do that and would need to do that is probably very small, if any.

Okay great thanks

Regarding port, Im not sure that would work as the device is quite picky about certs and such

[Boohooligan]

So, I'm seeing the setpoint_command in MQTT Explorer and the module checks in and updates but doesn't take it the new setpoint. I'm assuming it is supposed to clear. Also noticed it gives it in the format like 144.0 where everything else being reported seems to be just 144. Wondering if it being a float and not an int could be causing it to not take the command.

Also noticed the QoS is set to 0, which means (I think) the module might not always get it. I'm wondering if a different QoS needs to be in play for the set commands. I don't really understand how it worked the one time but I can't get it to work again.

As for changing grid state, I have no clue how to actually change it. I'd assume it's on the water heater settings somehow but I only have a mode/enter button and up and down arrows on the heater itself.

[starsoccer]

So, I'm seeing the setpoint_command in MQTT Explorer and the module checks in and updates but doesn't take it the new setpoint. I'm assuming it is supposed to clear. Also noticed it gives it in the format like 144.0 where everything else being reported seems to be just 144. Wondering if it being a float and not an int could be causing it to not take the command.

Also noticed the QoS is set to 0, which means (I think) the module might not always get it. I'm wondering if a different QoS needs to be in play for the set commands. I don't really understand how it worked the one time but I can't get it to work again.

As for changing grid state, I have no clue how to actually change it. I'd assume it's on the water heater settings somehow but I only have a mode/enter button and up and down arrows on the heater itself.

Hmm I can probably fixed that but it would be odd that it worked once. I don't think the setpoint would ever be cleared to be honest. Its probably just relayed once or something and then sits there.

As for grid state, usually atleast in my unit, you hold the mode/enter button for 10 seconds and then itbwill flash RA

[starsoccer]

So just to follow up it looks like using this, I can likely adjust the data sent to the temp set point, https://www.home-assistant.io/integrations/water_heater.mqtt/#temperature_command_template that would let me change it from a float hopefully.

Have you tried changing the mode on your unit from HA? That should be possible too and may be a bit easier to test, though in all honesty I may need to adjust that too as the modes dont map 1:1 as already mentioned and my mapping right now is just one way I think.

If your familiar enough to look at the MQTT payloads and see if they are float/string, etc.. You should also be able to adjust them and see what works for you unit. That would be very helpful for me.

I am not sure exactly when the unit updates, its very possible it takes ~5 mins for the update to take effect if its only checking in that frequently

[Boohooligan]

Can you tell me what the raw modes are? I tried the ones that HA provides but the system doesn't like that, and thinking about it now, yeah it doesn't make sense. Have to use a mode that the water heater understands but I don't know what the modes are as sent by the device.

[starsoccer]

You can see them all here, https://github.com/starsoccer/energysmartbridge/blob/master/energysmartbridge/src/Modules/MQTTModule.cs#L234, Basically the cases are what comes from the device, and the ha_mode is the mode we are sending to HA

I am going to try to decide this week if rewriting this whole thing is worth it or not so I can make improvements and changes easier.

[Boohooligan]

Oh wow so it actually reports just how it shows on the display, awesome.

So on my heater now shows lines instead of rA BUT it shows remote administration is on in HA.

I was able to change the mode fairly easily now, it seems reliable. I've changed modes 3 times now and it has changed with every check in. So it just seems the temperature setting now that is the problem.

Honestly i'm not sure it would be worth rewriting it all since it works. I think just getting the documentation in good shape (which I will start on today) will help people with this. Maybe just a few little tweaks to get it working nicely. You've done a ton.

[Boohooligan]

It may be worth reporting the actual mode in a separate sensor rather than the home assistant water heater modes. I know for me I'd rather it show the way it is on the water heater.

[starsoccer]

Can you clarify what you mean by it reports just as it shows?

yeah those lines I guess mean RA. Mine does that too. So if you disable RA and then try to change the mode does it still work?

Glad to hear mode changing is working. In regards to setting the temp, were you able to try setting it manually with mqtt without the decimal and see if it works?

I mean the code itself is relatively simple. I just want to add all the other sensors to HA and fix the above stuff as well, but the more I do, the more code it is to convert vs just rewriting to something I can more easily and then making improvements to it.

[Boohooligan]

For reporting just as it shows I mean a sensor that reports the raw mode: Efficiency, Hybrid, Electric, Vacation. Whatever the value from the water heater sends.

It seems to change mode regardless of the RA state.

I tried changing the temperature again with an int though and not only does it not do it, it stops taking mode change commands. It's still reporting data regularly but just will not accept a new mode or temperature. I'm considering rebooting it to try to clear whatever is blocked.

I'm using setpoint_command but I wonder if it should actually be something different?

I also don't understand how it worked once yesterday, unless there is a problem setting it above a certain number. I keep my water heater at 145 and the one time it worked it was a set point of 140.

[Boohooligan]

Ok, I got this all figured out now. Part of it was a network problem at my house that I managed to fix.

Grid state does not matter for changing temperature or mode. On or off, it still works.

The temperature is fine as a float but you cannot set it above 140F via this interface. When I was testing it was mostly all 141-145 and the one time it did work yesterday was when I set it to 140. So, I'd say this all works as intended.

I think the only feature I'd like to see at this point is just the raw mode state as a sensor. I dislike the modes that HA provides in the water heater integration.

[Boohooligan]

In a strange twist of fate though, I've lost my uppertemp_state and lowertemp_state sensors in HA. It's still reporting in MQTT, but HA doesn't automatically keep them there anymore. I added to my configuration.yaml and those new ones populate.