ngardiner / TWCManager

Control power delivered by a Tesla Wall Charger using two wires screwed into its RS-485 terminals.
The Unlicense
129 stars 55 forks source link

"Low Quality Grid Detected" and downward spikes during charging on 3-phase (at low charge rate) #298

Open bikeymouse opened 3 years ago

bikeymouse commented 3 years ago

Hi, I'm only using TWC now for a couple of weeks to charge with tracking solar, but I noticed some strange behavior I don't have when charing on full power. I'm wondering if it is "normal" or even healthy for my car/charger.

I'm measuring every 10 seconds the Power, Amps and Voltage on the 3 phases to the TWC, and I see that when charging at (3x) 6A or 7A the car regularly seems to switch off 1 or sometimes even 2 phases. It doesn't seem to correlate with the Voltages, as these are within tolerance (between 227-234V) and the phase that is switched off (L2) is not the one with the lowest average voltage.

I even had a case where it didn't start charging on 1 phase until I stopped and re-started the charging from the Tesla App and started off again with 16A.

This is on a 2019 Tesla Model 3. Is this something others are seeing as well?

I'm a bit afraid that this high-power switching is not really good for the life time of the on-boad convertors. So I hope there is a way to prevent this from happening (other then charging at > 3 x 10-16 Amps).

Charging with spikes
ngardiner commented 3 years ago

Hi @bikeymouse,

If this is happening, it's not through any of our doing. We have no way to instruct TWCs to do anything with any individual phase, I wish we did. If you were to power down the TWC, set the rotary switch to the same number of amps we're offering and measure again, you should see the same thing - as that is all we do. We just instruct the TWC as to how many amps it can offer.

Are you sure it's not your instrumentation?

bikeymouse commented 3 years ago

Hi, Yeah, I know it is not something TWCManager is instructing the TWC to do, so it probably is going to happen also if the rotary switch is set to these values.

It is of course possible, but I think that it is not my instrumentation, as I see a clear relation with the number of Amps offered. It happens a lot at 6/7 Amps, gets a lot less at 8/9 Amps and I haven't seen this at > 10 Amps.

Does anybody else also measure their 3 phases to the TWC like I do and can confirm/deny if they see this happening too?

bikeymouse commented 3 years ago

I did some more cross checking to make sure that these are not measurement equipment errors.

Luckily I have an independent reading from my DSMR-meter that measures the power consumption or return to my energy provider, so I could combine the data.

If you look at the graph below, you can see that on this sunny day, there is a direct relation between the downward spike on the power to the TWC and the returned power to the energy-net:

TWC spiking

Charging starts at about 10:07 hour. Then there are a lot of downward spikes (caused by switching of Phase 2 which I can see in detailed current graphs as posted earlier). At about 10:58 hour it decides to completely switch off Phase 2, so it keeps tracking excess solar with reduced power to about 3,5 Kw (2 x 7A or 2 x 8 A). Phase-2 then returns about 1 Kwh (4/5 Amps) to the net.

So to me this is defintive proof that the TWC/car is actually spiking on 1 or 2 phases. Don't think that is good for the actuators. Love to hear if anyone else is doing this kind of measuring with similar results?

Edit: I know noticed in the car that it reports “low quality grit” and that I need to reconnect.

This should not be caused by too low Voltages as these are alway within acceptable range (223V - 240V), and as said before I never have any problems charging at higher Amps then 10A.

bikeymouse commented 3 years ago

Another check why I don't think the " low quality grid" error - that only occurs when charging with low power (<8 A) - is correct.

The last error occurred at 10:59. Then the car completely switched of charging at phase-2.

However If I look at the measurements around that time from 2 independent meters (one DSMR-meter from the energy-company measuring the grid, the other my own Modbus meter measuring the power to the TWC), they both report the same voltages, all nicely within all tolerances.

Top graph: showing current switching spikes on phase-2 by the TWC/car, finally completely shutting down, resulting in current being returned to the grid.

Bottom graph: showing voltages on phase-2 both from the mains and the TWC, with no spikes. Voltages between 235V - 239V (so +/- 1% ) while charging and between 239V - 242V while returning green power to the grid. TWC

This error never occurred last 2 years when I was always charging at 11Kw / 16Amps. On any charger. Even with a grany charger with 212 Volts. But now with TWC Manager and charging with < 7 Amps it is. So I find it hard to believe the error is actually something in the grid or high voltage wiring or instrumentation.

ngardiner commented 3 years ago

This should not be caused by too low Voltages as these are alway within acceptable range (223V - 240V), and as said before I never have any problems charging at higher Amps then 10A.

From the explanations online, this error doesn't reflect that voltages are too low - it reflects that the vehicle has measured excessive voltage drop as it increases power draw.

ngardiner commented 3 years ago

Have you tried setting the DIP switch to a lower amperage and seeing if it still happens?

bikeymouse commented 3 years ago

This should not be caused by too low Voltages as these are alway within acceptable range (223V - 240V), and as said before I never have any problems charging at higher Amps then 10A.

From the explanations online, this error doesn't reflect that voltages are too low - it reflects that the vehicle has measured excessive voltage drop as it increases power draw.

Agreed, but if that would happen, why doesn't these voltage drops show in the Voltage graphs above? And why does this only happen at 6-8 Amps? According to Ohms law, I would expect this to happen when switching e.g. from 0A to 16Amps, but that has never happened. The only thing I can think off that if other big consumers in the house, like a dryer, kick in, this would have a relative bigger impact when using low power. But still I'd expect the car would only trigger on absolute voltage drops.

bikeymouse commented 3 years ago

Have you tried setting the DIP switch to a lower amperage and seeing if it still happens?

Yes, good suggestion. I have just done that. Not having done any charging yet, but I do see something in the logs: I have to set TWC to be slave now, so using fakemaster = 2 Now I see these errors in the log:

Fakemaster=2 loglevel=20
10:02:38 TWCManager 20 ***UNKNOWN MESSAGE from master: FD EC 60 90 04 05 03 02 00 00 00 00 00 00 00 EA
10:02:39 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:39 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:40 TWCManager 20 ***UNKNOWN MESSAGE from master: FD EE 60 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 DE
10:02:40 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:41 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:41 TWCManager 20 ***UNKNOWN MESSAGE from master: FD EF 60 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 DF
10:02:42 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:42 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:43 TWCManager 20 ***UNKNOWN MESSAGE from master: FD F1 60 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 E1
10:02:43 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:44 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:44 TWCManager 20 ***UNKNOWN MESSAGE from master: FD ED 60 90 41 31 38 49 30 30 38 36 30 39 30 00 00 00 00 37
10:02:45 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:45 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:46 TWCManager 20 VRS 6090: 140kWh 244V 239V 235V
10:02:46 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
10:02:47 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
...repeat...

When I set loglevel=10 this is the result:

Fakemaster=2 loglevel=10
09:58:21 TWCManager 20 ***UNKNOWN MESSAGE from master: FD F1 60 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 E1
09:58:21 TWCManager 9 Ignoring byte FD between messages.
09:58:21 TWCManager 12 Rx@: (FD) FB E0 60 90 77 77 02 05 00 00 00 00 00 00 00 C5
09:58:21 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
09:58:21 RS485      12 Tx@: C0 FD E0 77 77 60 90 07 0F A0 0E 88 00 00 0A C0
09:58:21 TWCManager 9 Ignoring byte FC between messages.
09:58:22 TWCManager 12 Rx@: (FC) FB E0 60 90 77 77 02 05 00 00 00 00 00 00 00 C5
09:58:22 TWCManager 20 Master heartbeat contains error 5: 02 05 00 00 00 00 00 00 00
09:58:22 RS485      12 Tx@: C0 FD E0 77 77 60 90 07 0F A0 0E 88 00 00 0A C0
09:58:22 TWCManager 9 Ignoring byte FC between messages.
09:58:22 TWCManager 12 Rx@: (FC) FD ED 60 90 41 31 38 49 30 30 38 36 30 39 30 00 00 00 00 37
09:58:22 TWCManager 20 ***UNKNOWN MESSAGE from master: FD ED 60 90 41 31 38 49 30 30 38 36 30 39 30 00 00 00 00 37
etc.

In normal use (with the dipswitch to "F") I never get any errors. Also if I set TWC as a master again I get no errors, except of course that it doesn't find a slave and only gets link-ready messages from the TWC.

Fakemaster = 0

09:48:26 TWCManager 20 TWC Manager starting as fake Master with id 7777 and sign 77
09:48:26 TWCManager 12 Rx@: () FC E1 60 90 53 00 00 00 00 00 00 00 00 00 00 24
09:48:26 TWCManager 20 Master TWC 6090 Linkready1.  Sign: 53
09:48:26 TWCManager 12 Rx@: (3C 00) FC E1 60 90 53 00 00 00 00 00 00 00 00 00 00 24
09:48:26 TWCManager 20 Master TWC 6090 Linkready1.  Sign: 53
09:48:26 TWCManager 12 Rx@: (3C 00) FC E1 60 90 53 00 00 00 00 00 00 00 00 00 00 24
09:48:26 TWCManager 20 Master TWC 6090 Linkready1.  Sign: 53

Any idea what the "Unknown messages" are and if this indicates something wrong in the communication?

ngardiner commented 3 years ago

Agreed, but if that would happen, why doesn't these voltage drops show in the Voltage graphs above? And why does this only happen at 6-8 Amps? According to Ohms law, I would expect this to happen when switching e.g. from 0A to 16Amps, but that has never happened.

All I can do is refer back to my earlier answer on this subject - these are interactions between the vehicle and the TWC, and you need to ask Tesla these questions. The performance of the TWC at lower amperage in your environment just isn't a subject that pertains to the development of TWCManager, no part of the protocol that we speak with the TWC pertains to the enabling or disabling of the input phases or the display of the message on your vehicle, all we do is electronically change the allocation of amps the way you can do mechanically with the rotary switch.

The questions you are asking above are not for the developers of TWCManager to answer, and while I'm okay for you to keep this subject to this issue, any further issues on this topic will get closed and redirected here from here on in, because I can't let the project get bogged down with support for hardware installations when our very limited time needs to be put into adding software features.

ngardiner commented 3 years ago

Any idea what the "Unknown messages" are and if this indicates something wrong in the communication?

fakeMaster = 2 hasn't been developed significantly in 2 years. It was originally implemented as a sniffer to observe messages from the Master to a Slave and was used to build the original parsers so we could reverse engineer the TWC protocols. It's served its purpose in that it decoded a lot of the protocol long ago, but there's plenty it can't parse - it (probably, I can't remember) wouldn't be able to parse things like requesting the VIN of a connected vehicle or requesting volts per phase or lifetime kWh, because these things were found by the analysis of firmware on a TWC Device by CDragon, the original TWCManager developer, and so these never needed a parser written.

That said, rather than removing it (which I would have been tempted to do), it will get updated someday in the possibly near future, as fakeMaster = 2 functionality is seeing something of a revival in the form of the Dummy module, which allows us to pretend to be a slave in order to test TWCManager entirely within software without the need for a hardware TWC to do testing. This has led to the introduction of a new parser in the form of the TWCProtocol module which is currently only used by the Dummy module,, but which once it becomes smart enough as a decoder for TWC messages, could be swapped in for the current fakeMaster = 2 parser and will probably then be extended.

It is difficult from a development perspective to run a TWC in fakeMaster = 2 mode for a period of time as few people have spare cars and TWCs to monitor and it disrupts being able to use TWCManager as it was intended to charge off of generated energy, so I'm not particularly motivated to do it, but it would help a lot with making the Dummy module real-world ready, which would in turn make our test suite even more real-world like, but remember that we do these things in our spare time and we don't have a lot of it, so it's firmly in the future that it may improve.

bikeymouse commented 3 years ago

Agreed, but if that would happen, why doesn't these voltage drops show in the Voltage graphs above? And why does this only happen at 6-8 Amps? According to Ohms law, I would expect this to happen when switching e.g. from 0A to 16Amps, but that has never happened.

All I can do is refer back to my earlier answer on this subject - these are interactions between the vehicle and the TWC, and you need to ask Tesla these questions. The performance of the TWC at lower amperage in your environment just isn't a subject that pertains to the development of TWCManager, no part of the protocol that we speak with the TWC pertains to the enabling or disabling of the input phases or the display of the message on your vehicle, all we do is electronically change the allocation of amps the way you can do mechanically with the rotary switch.

The questions you are asking above are not for the developers of TWCManager to answer, and while I'm okay for you to keep this subject to this issue, any further issues on this topic will get closed and redirected here from here on in, because I can't let the project get bogged down with support for hardware installations when our very limited time needs to be put into adding software features.

Ok, fair enough. I was just trying to give all the info to make sure that this not TWCManager related but something else.

bikeymouse commented 3 years ago

Any idea what the "Unknown messages" are and if this indicates something wrong in the communication?

fakeMaster = 2 hasn't been developed significantly in 2 years.

Ok, that is something as a relieve as I was afraid I had to resolve communication/RS-485 issues ;-) As I normally don't get any errors (with fake master = 1) and there are no CRC-errors, I think I can assume that part is working fine.

bikeymouse commented 3 years ago

Some update about setting the TWC to lower Amps:

I started by setting the switch in the TWC at "1" which, according to the table in the TWC, corresponds to max 6Amps.

Guess what: now the car doesn't want to charge at all. It gives an error saying there are not enough Amps available and I should move to another charger or remove other cars. Strangely enough the TWC also goes into error and starts flashing a slow red LED.

So I switched the TWC off and set the switch to the "2" position. That corresponds to max 8 Amps. After starting up and connecting the car again it immediately starts charging with 8Amps. And until now now downward spikes. I'll post an update if something noticeable happens.

ngardiner commented 3 years ago

It sounds like the correct configuration for your TWC would be minAmpsPerTWC=8, since you were seeing issues at 6/7 amps and your TWC refuses to charge at 6A on the rotary switch. If it won't charge at 6A when mechanically set (not sure why, never seen that before, is it voltage related?) it's probably not a good idea for us to instruct it to charge that low.

This is a first though - never had a report of anyone having issues charging at any mechanically selectable amperage on a TWC before.

bikeymouse commented 3 years ago

Ok have charged for an our with a fixed setting of 8Amps in the TWC without problems. Then changed the TWC back to being a slave and set TWCManager minAmpsPerTWC to 8. That also worked (as expected) without issues, nicely tracking solar between 8 - 10 Amps.

One thing I noticed is that when not charging and sitting in the car with the charger connected, is that the Amps offered alternated between 8 and 16Amps. That is probably "by design", with TWCManager trying to start charging and "spiking" it, not being aware of this use case. After unselecting "reactively" checkbox for the spike Amps setting in the debug-setting page in the front-end, it didn't occur anymore. This didn't negatively affect the charging and tracking solar later.

So the conclusion indeed is that 8Amps is indeed the minimum setting for my 2019 Model 3 with my TWC.

An explanation that no other people have reported this before with TWCManager, might be that I only noticed the spiking, or one phase being switched off completely, by looking at my Modbus meter-logging (until recently I didn't see or notice the "low quality grid" error).

Most people probably rely on the Power and Amps figures reported by TWCManager, however these do not reflect if one phase is throttling or completely switched off: the power is always calculated as Amps x 3 phases x Voltage.

ngardiner commented 3 years ago

An explanation that no other people have reported this before with TWCManager, might be that I only noticed the spiking, or one phase being switched off completely, by looking at my Modbus meter-logging (until recently I didn't see or notice the "low quality grid" error).

Strange conclusion - given nobody has reported charge instability or the low quality grid error before I don't think that is going to mean everyone is affected and don't know it, I suspect it simply means your install is affected below 8A and you'll need to set that as your minimum, or call Tesla to understand why it can't charge below 8A.

ngardiner commented 3 years ago

Sorry, that close was accidental.

VIDGuide commented 3 years ago

FWIW, I was having a phase issue with mine when I got my car last week. Electricians (who did my Solar upgrade) had wired it up, but I hadn't paid much attention to it until I needed it. Plugged into the car, and the car shows a warning that one phase is missing, charging is slowed. Checked TWCManager and sure enough, L2 was 0V.

Every now and then if I did breaker resets on the TWC, I'd see all 3 phases in TWCManager, so I'd try to charge, and then almost immediately it would go into voltage spike errors and stop. At that point I'd see I was back to 0V on L2.

Called them back, but they still haven't showed, so I got my hands dirty and metered it out. Turns out they literally hadn't done up all the screws on the 3-phase breaker. L2 was sitting loose in the socket. Once I tightened it all up, it's been rock solid since.

My conclusion was that flipping the breaker would knock the wire into making enough contact to register voltage, but as soon as the car started pulling current through it, it wasn't sufficient and it lost contact/dropped it. The error lights on the TWC basically indicated voltage drop/spike detected.

So it is possible your wiring isn't happy at the higher amperage, and this is how it's showing up.

bikeymouse commented 3 years ago

Thanks for your info. Yeah, most logical would be something in the wiring. However the thing is, I have been charging with exactly the same wiring over a year now always at 16Amps. Only shortly after I switched over to use TWCManager and charge at lower Amps I see this behavior. And it seems that it happens more at low Amps. At > 10 Amps I haven't even seen it happening yet. While, if it was due to higher resistance in the wiring, that should be completely the other way around. So of course it can be a freak coincidence that this happened just when starting to use TWCManager, as my TWC/Car-combination seems to be the only one having this (or recording this...). FYI: the error "low quality grid detected" only happened twice, while the spiking happens way more often.

So I'll have to investigate further to see what is happening here. So to be sure I think I'll have to set the TWC back to be master and at 16Amps and then see if it stops spiking. If so, then I think I can conclude that it is not the high voltage wiring, right?

ngardiner commented 3 years ago

So I'll have to investigate further to see what is happening here. So to be sure I think I'll have to set the TWC back to be master and at 16Amps and then see if it stops spiking. If so, then I think I can conclude that it is not the high voltage wiring, right?

This is the sort of question best left to an electrician or Tesla. The fact that we don't have scores of users here seeing what you're seeing (regardless of your issue being the single only issue pinned for visibility) should be further confirmation to you that you need your install checked if you're experiencing strange behaviour. If you simply disconnect TWCManager, set your rotary switch to 6A and then ask your installer why it doesn't charge at 6A, you'd likely solve yourself a lot of time, frustration and potential hazard. I'm not sure how long you've had it installed for but this is more than likely covered by warranty or follow up service, but won't be forever, so the longer you play with the settings, the more it could potentially cost to fix.