OpenEVSE / openevse_esp32_firmware

OpenEVSE V4 WiFi gateway using ESP32
176 stars 118 forks source link

OpenEVSE constantly dropping Wifi #841

Open jeremyakers opened 6 months ago

jeremyakers commented 6 months ago

My OpenEVSE unit loses it's wifi connection constantly. I have to flip the breaker off and back on daily, sometimes multiple times per day, to get the unit to reconnect.

This has been ongoing since I purchased the unit but has become more of a problem as I rely on the MQTT connection for self production. When the unit loses wifi, it also loses the MQTT connection and the unit continues charging at whatever rate the last MQTT message commanded. So we frequently find that the unit has failed to charge our vehicle at all after a nice sunny day because the unit dropped connection overnight when the sun was not shining.

I've checked and the unit does not broadcast a "OpenEVSE_XXXX" network after losing connection. It's almost as if the wifi radio has just turned off. Turning the breaker off and back on fixes it everytime and no other wifi devices in the garage suffer any wifi signal issues. The wifi access point is on the other side of the garage wall just a few feet from where the EVSE is installed so wifi signal at the EVSE is quite strong.

Dicion commented 5 months ago

Lets check the WiFi signal strength of your device. Lets see what it's seeing.

On the network page, hover over the signal icon and see what it says. Lets see if it's reporting too low (or too high) a signal.

image

Mine says -39dbm in the above image. What does yours say?

According to this random website I found, ESP32 signal strength quality is as follows:

image

jeremyakers commented 5 months ago

Screenshot from 2024-06-19 21-11-32-cropped

-52. So that would be "Very good" according to your chart.

Also worth noting I have several other ESP32 devices in the same garage, plus a wifi water softener, plus a wifi garage door opener, etc. None of these other devices ever lose connection.

Additionally: If wifi strength was the problem resetting the breaker would not likely reliably fix it: as it would be just as unlikely to connect to weak wifi signal at boot-up as any other time. But that's not what happens: it connects to the wifi perfectly fine every single time it boots up. Then minutes later, or maybe hours later, or maybe days later, it drops connection and will never reconnect on it's own unless I power cycle, and then like clockwork the power cycle fixes it every time, all the time, but only temporarily.

Dicion commented 5 months ago

Yup, that's quite odd. Have you tried reloading the ESP32 firmware from scratch? Seeing if that makes a difference?

If you've reloaded the firmware and tried a full reset, then it might be an issue with the module itself. If it were me, I'd replace the ESP32 board and see if it makes a difference.

I don't have the LCD module, I have the older Wifi kit module, and mine's been rock solid.

jeremyakers commented 5 months ago

Funny enough, Chris at OpenEVSE sent me a new Wifi board that I just replaced last night right after our exchange (This EVSE was the $600 version that I purchased fully assembled, it was not a kit I put together myself) And so far it's behaving exactly the same way. It ran fine for about 8 hours and then dropped off the wifi until I walked out and did a full power reset on the EVSE. So it seems unlikely to be an issue with the Wifi module hardware itself.

I notice on my other ESP32 devices, if I manually reboot by wifi router or do anything to make the Wifi drop: They always switch to AP mode where they broadcast their own SSID until they are able to reconnect to my wifi network. When the EVSE drops off the wifi though: It does nothing. Which is why I suspect the wifi radio is being turned off, or maybe losing power internally?

Oh, and another thing to mention: Prior to the OpenEVSE I had a GoPlug EVSE in this exact same mounting location. I'm pretty sure the GoPlug uses the same ESP32 based wifi module just with slightly different firmware (It looks like an older version of OpenEVSE firmware to be honest): And it never had any connectivity issues at all. It ran perfectly fine in this location for several years until it started having overheating issues from a bad component (relay/contactor I think) and I replaced it with the OpenEVSE.

So it's either a hardware problem deeper in the EVSE: IE: Maybe the board that supplies power to the ESP32 is the issue? Or it's a bug in the firmware that seems to be specific to the way I'm using it with MQTT+Solar Divert...

chris1howell commented 5 months ago

The GoPlug was a stock 2017/2018 OpenEVSE station with different artwork and only basic features, the advanced features simply hidden in firmware. The quick start guides and their website was a copy from OpenEVSE. The Goplug had ESP8266 WiFi just like the OpenEVSE stations of the same period. There were no differences in hardware or core firmware. OpenEVSE manufactured the first 275ish units for GoPlug and provided GoPlug components for 225 more.

The issue does not sound like hardware. We sent a replacement module to rule it out as a possibility.

On Thu, Jun 20, 2024, 1:05 PM Jeremy Akers @.***> wrote:

Funny enough, Chris at OpenEVSE sent me a new Wifi board that I just replaced last night right after our exchange (This EVSE was the $600 version that I purchased fully assembled, it was not a kit I put together myself) And so far it's behaving exactly the same way. It ran fine for about 8 hours and then dropped off the wifi until I walked out and did a full power reset on the EVSE. So it seems unlikely to be an issue with the Wifi module hardware itself.

I notice on my other ESP32 devices, if I manually reboot by wifi router or do anything to make the Wifi drop: They always switch to AP mode where they broadcast their own SSID until they are able to reconnect to my wifi network. When the EVSE drops off the wifi though: It does nothing. Which is why I suspect the wifi radio is being turned off, or maybe losing power internally?

Oh, and another thing to mention: Prior to the OpenEVSE I had a GoPlug EVSE in this exact same mounting location. I'm pretty sure the GoPlug uses the same ESP32 based wifi module just with slightly different firmware (It looks like an older version of OpenEVSE firmware to be honest): And it never had any connectivity issues at all. It ran perfectly fine in this location for several years until it started having overheating issues from a bad component (relay/contactor I think) and I replaced it with the OpenEVSE.

So it's either a hardware problem deeper in the EVSE: IE: Maybe the board that supplies power to the ESP32 is the issue? Or it's a bug in the firmware that seems to be specific to the way I'm using it with MQTT+Solar Divert...

— Reply to this email directly, view it on GitHub https://github.com/OpenEVSE/openevse_esp32_firmware/issues/841#issuecomment-2181157361, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN5QH4LJOXV6B3RAHZTOIDZIMDU5AVCNFSM6AAAAABHUPER6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBRGE2TOMZWGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Dicion commented 5 months ago

Or it's a bug in the firmware that seems to be specific to the way I'm using it with MQTT+Solar Divert...

Have you tried disabling this and seeing if it behaves with it off? If you can do that, and it stays solid, that could help isolate it to where it is happening. Maybe it could also be related to just the amount of data you're sending constantly due to that? Unfortunately without your exact setup, there isn't much anyone else can do. I haven't seen mass reports of WiFi issues, which means it's something unique to your setup or configuration that's likely causing it.

Ideally you'd hook up to the debug serial port and use it to log what it says/does when this occurs, or even enable some enhanced debugging and do the same, but I don't think that is realistic to ask most users to do, as the first requires soldering, the second requires a custom build with enhanced debugging enabled.

I think I'm going to order a new TFT model, and give my current one to a friend as a gift (So I can charge at his house for free :P). If this still isn't resolved by the time I get it and get it setup, I'll see about emulating your setup and see if I can reproduce it.

jeremyakers commented 5 months ago

I'm going to try that next. It's just that the whole point of purchasing the OpenEVSE over the cheaper alternatives was specifically for the solar divert functionality. If I have to disable that feature it kind of defeats the point of making the investment into this product. I was using the solar divert functionality on the GoPlug without any issue as well and knowing that the GoPlug was based on OpenEVSE to some extent I assumed it would work just as well.

KipK commented 5 months ago

Some thing that could be investigated as a culprit could be the new changes made this year on mqtt. Lot of data has been added to the mqtt like config and some other values. As I was out of the project for a while because of personal life changes, I haven't followed the code changes, but it could be a memory or overflow issue.

I hadn't encountered wifi drops, but some mqtt drops, with no way of reconnecting to mqtt server without rebooting the wifi module. Unfortunately I couldn't debug this as I still have no time to investigate :(

Dicion commented 5 months ago

@jeremyakers I understand, this is more a troubleshooting step to see if you can confirm that the MQTT and/or Solar Divert is what's causing the issue. If it can get narrowed down to one feature causing a memory or overflow issue, then it can be found and fixed.

@KipK If you know when/what build has the MQTT changes, perhaps @jeremyakers can try one prior to that as well, see if it works without causing the issues. I can look as well, see if I can identify a candidate for them to try.

KipK commented 5 months ago

Here are the related mqtt commits starting from may 2023:

https://github.com/OpenEVSE/openevse_esp32_firmware/pulls?q=is%3Apr+is%3Aclosed++mqtt

jeremyakers commented 5 months ago

Well since I completely disabled MQTT and self production I've had to reset the unit twice now to reconnect it to Wifi... So I guess it's not MQTT after all. I am not sure what else to try at this point?

jeremyakers commented 5 months ago

Actually now it's doing something different. The unit appears to be rebooting at random times:

image

If you look at this history: The car was unplugged at 5:50pm and we haven't touched the unit since then (My breaker resets to get it back on Wifi were prior to unplugging the car) so I'm not sure what all these log entries are for. Normally I don't see any new log entries in the history log when the car isn't plugged in.

jeremyakers commented 5 months ago

Another update: I just noticed something that I hadn't noticed before.

The 12v battery on one of our cars got low and we had to jump start it. The main battery was at 90% so to help ensure the 12v battery was fully charged I plugged it in and manually set the charging current to 6amps. The car showed an estimated time to charge to 100% of around 6 hours. My hope here was to give the DC/DC converter plenty of time to recharge the 12v battery since it mainly only charges the 12v battery when the main battery is charging and I knew at the normal 40 amp setting it would be charged in less than an hour.

So anyway: Almost immediately after setting the unit to 6 amps: It dropped off the wifi again. I walked out to reset the breaker and I noticed on the display that it had reverted back to it's 40 amp setting. So I tripped the breaker, waited for everything to stop, turned the breaker back on, logged back into the EVSE and reset the current back to 6 amps again. I verified the remaining charge time on the car was now showing nearly 6 hours and I went back to my normal day job.

I checked the monitoring screen of the EVSE a couple more times over the next 15 minutes to make sure it was still working and then started doing work Zoom meetings and calls.

About 30 minutes later I got a notification from the car that charging had been complete. I immediately try to load the EVSE web page again but it won't load: It dropped off wifi again. I walk out to the unit and notice that it had again reset itself back to 40 amps.

So I think the issue here is deeper than the Wifi module. I don't think the wifi module losing wifi connection would cause the unit to reset the charging current, would it? Seems like the actual OpenEVSE board is resetting itself (Which may be what's causing the wifi module to drop connection)

I never noticed this before because I very rarely manually lower the charging current.

chris1howell commented 5 months ago

All of the intelligence comes from the WiFi module. The OpenEVSE controller manages safety only. The connection between is a simple serial interface.

Does the vehicle stop charging when WiFi drops? Or does the charge complete? Can you start a new charge session after WiFi drops?

On Mon, Jun 24, 2024, 2:34 PM Jeremy Akers @.***> wrote:

Another update: I just noticed something that I hadn't noticed before.

The 12v battery on one of our cars got low and we had to jump start it. The main battery was at 90% so to help ensure the 12v battery was fully charged I plugged it in and manually set the charging current to 6amps. The car showed an estimated time to charge to 100% of around 6 hours. My hope here was to give the DC/DC converter plenty of time to recharge the 12v battery since it mainly only charges the 12v battery when the main battery is charging and I knew at the normal 40 amp setting it would be charged in less than an hour.

So anyway: Almost immediately after setting the unit to 6 amps: It dropped off the wifi again. I walked out to reset the breaker and I noticed on the display that it had reverted back to it's 40 amp setting. So I tripped the breaker, waited for everything to stop, turned the breaker back on, logged back into the EVSE and reset the current back to 6 amps again. I verified the remaining charge time on the car was now showing nearly 6 hours and I went back to my normal day job.

I checked the monitoring screen of the EVSE a couple more times over the next 15 minutes to make sure it was still working and then started doing work Zoom meetings and calls.

About 30 minutes later I got a notification from the car that charging had been complete. I immediately try to load the EVSE web page again but it won't load: It dropped off wifi again. I walk out to the unit and notice that it had again reset itself back to 40 amps.

So I think the issue here is deeper than the Wifi module. I don't think the wifi module losing wifi connection would cause the unit to reset the charging current, would it? Seems like the actual OpenEVSE board is resetting itself (Which may be what's causing the wifi module to drop connection)

I never noticed this before because I very rarely manually lower the charging current.

— Reply to this email directly, view it on GitHub https://github.com/OpenEVSE/openevse_esp32_firmware/issues/841#issuecomment-2187173190, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN5QH5VBOE6JE4YYVYXADLZJBRDNAVCNFSM6AAAAABHUPER6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGE3TGMJZGA . You are receiving this because you commented.Message ID: @.***>

jeremyakers commented 5 months ago

"Does the vehicle stop charging when WiFi drops?"

I'm not sure: If the charger stops for just a few seconds (Like when I manually trip the breaker to reset the wifi connection) my car doesn't send any kind of notification... so it's entirely possible that it's stopping and then restarting so quickly that it doesn't send the "charge interrupted" notification.

"Or does the charge complete?"

The charge does eventually complete. That's what happened here: I set it manually to 6 amps but the EVSE kept resetting itself back to 40 amps. So the charge completed in around an hour when it should have taken 5+ hours.

"Can you start a new charge session after WiFi drops?"

Yes, if I plug a vehicle in after the wifi drops the EVSE charges. I just can't access the interface to change the charging parameters.

chris1howell commented 5 months ago

It does not sound like the controller is at all involved. If there were some sort of power glitch both WiFi and controller would come back online just like a breaker reset. The controller has no capability to control WiFi, it just reports it's status and accepts simple commands.

The issue sounds like a classic software lockup. Something is crashing the WiFi making it unresponsive until power cycles.

Now the tricky part is figuring out what is going on, many many many hundreds of people are using MQTT and Solar Divert is probably the most popular feature. It does not seem others are reporting similar issues.

On Mon, Jun 24, 2024, 3:44 PM Jeremy Akers @.***> wrote:

"Does the vehicle stop charging when WiFi drops?"

I'm not sure: If the charger stops for just a few seconds (Like when I manually trip the breaker to reset the wifi connection) my car doesn't send any kind of notification... so it's entirely possible that it's stopping and then restarting so quickly that it doesn't send the "charge interrupted" notification.

"Or does the charge complete?"

The charge does eventually complete. That's what happened here: I set it manually to 6 amps but the EVSE kept resetting itself back to 40 amps. So the charge completed in around an hour when it should have taken 5+ hours.

"Can you start a new charge session after WiFi drops?"

Yes, if I plug a vehicle in after the wifi drops the EVSE charges. I just can't access the interface to change the charging parameters.

— Reply to this email directly, view it on GitHub https://github.com/OpenEVSE/openevse_esp32_firmware/issues/841#issuecomment-2187282822, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN5QH3TPVEZXXCEOFNL6FLZJBZJZAVCNFSM6AAAAABHUPER6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGI4DEOBSGI . You are receiving this because you commented.Message ID: @.***>

jeremyakers commented 5 months ago

many many many hundreds of people are using MQTT and Solar Divert is probably the most popular feature.

Right, so if you look back in this thread: I completely disabled MQTT and Solar Divert to rule that out. So it appears something else is causing it... But I'm not really sure what else to check at this point?

Dicion commented 5 months ago

Question about your wifi access point.

What make/model/version of Wifi is it? What type of security are you running on it?

Do you have the ability to create an additional SSID with different settings for testing?

jeremyakers commented 5 months ago

What make/model/version of Wifi is it? What type of security are you running on it?

Netgear WAC104 - image

image

Do you have the ability to create an additional SSID with different settings for testing?

Are you suggesting I put another WAP next to this one and connect to that one instead? I suppose I could but I'm not entirely clear on how that would help?

Dicion commented 5 months ago

Are you suggesting I put another WAP next to this one and connect to that one instead? I suppose I could but I'm not entirely clear on how that would help?

No, some Access points allow you to create X amount of SSID's on them that can all work the same, or differently, with different settings, subnets, vlans, etc. It does not look like your AP above supports that. It also appears to only be using WPA2, which should be fine.

Newer APs running Wifi 6 require WPA3, which is not backwards compatible to a lot of devices, and will break communications to older devices that do not support WPA3, but that does not appear to be an issue here.

I just wanted to make sure we weren't in some setup where that issue might be coming into play at all.

You COULD try to disable the 'Enable 20/40mhz coexistence' setting and see if it makes a difference. What this setting does is the following:

What [20/40 MHz Coexistence] setting does is it allows the 2.4 GHz radio to use the full 40 MHz bandwidth, (and communicate with both 20 MHz and 40 MHz bandwidth clients just fine), unless it encounters another AP which is using a nearby channel on the 2.4 GHz band, and interference is inevitable. When this happens, both APs will fall back to 20 MHz only, allowing them to both coexist. (This also means that 40 MHz capable client devices will also fall back to 20 MHz, since they cannot find a 40 MHz bandwidth channel).

Basically, with this setting enabled, the AP may be changing the channel widths on the fly if/when it sees another AP in the area, which SHOULDN'T be an issue, but at this point, I recommend you try everything.

So try turning that setting off and see how it performs. It may work, it may break it completely, it may change nothing. Worst case, you can just turn it back on.

jeremyakers commented 5 months ago

I disabled that feature this morning. We'll see how it goes. But given the GoPlug worked just fine before, and I have other ESP32 based devices (Konnected boards) in the same garage I'm skeptical that changing anything on the Wifi side will help.

I'm wondering if turning off the MQTT is really enough. Does turning MQTT off actually completely remove the MQTT config or could there still be pieces of that config being read at bootup that could affect the units behavior? Should I do a factory reset to ensure that the MQTT settings are fully wiped?

jeremyakers commented 5 months ago

Yeah disabling that feature didn't do anything. Still dropping. I'm going to try full factory reset next.

jamarju commented 3 days ago

@jeremyakers Did you get to resolve this? I have the same problem: charger starts charging then drops WiFi after a few hours. I have the TFT version on firmware version 5.1.2. I do use MQTT for both shaper and solar divert.

One thing I noticed is the clock in the display keeps updating after the WiFi is lost so I guess the firmware is still running somehow, just dropping the WiFi.