Add support for Victron SmartShunts through VE.Direct to read the SOC and other Battery details

[swingstate]

Is your feature request related to a problem? Please describe.

Feature Request: I shared my finding in #185 and suggest to look into adding support for at least one dedicated Smartshunt through Ve.Direct.

Describe the solution you'd like

The Shunt should use dedicated GPIO Pins (needs 2) to interface with the ESP board. I didn't look into whether with the current setup GPIO Pins are still available. This development should also consider the potential future support for a second MPPT, hence the "PIN Strategy" should be set up carefully.

The Webinterface should allow to switch between CAN Bus and VE.Direct SOC reading.

Describe alternatives you've considered

Alternative (cheaper, a little bit less accurate) to read the SOC from Batteries who do not provide a CAN Bus interface would be the Support for Shelly UNI.

Additional context

No response

[helgeerbe]

@swingstate I think it is quite easy to extend my ve.direct frame handler to support also Victron's shunt and use ist as an alternative battery monitor.

I don't own a shunt, so I need someone for testing. One other thing is my time. I try to fix a nasty power limiter issue this weekend. So it will take some time before I can start.

[swingstate]

No issue, and no pressure. I am closely following the bug hunt :) I am available to test any code with my shunt. Just let me know when you have capacity and want to look deeper into this. I'll then prepare an OpenDTU board for testing.

[spcqike]

Hello all

Just out of curiosity, would it be possible to implement an own shunt?

Like it looks like it’s possible to build your own Victron shunt https://github.com/kscholty/SmartShuntINA226

Maybe it’s possible to directly include it to opendtu? Would this be useful?

[madmartin]

@swingstate can you record a sample of the text mode output of a victron smartshunt and copy it here? I plan to implement that.

[spcqike]

@madmartin i'm not sure, if it's of any help, but as the linked project claimes to be compatible with victron, here is its output

several of these:

PID     0xa389
V       0
I       6
P       0
CE      0
SOC     0
TTG     -1
Alarm   OFF
Relay   OFF
AR      0
AR      0
BMV     INR226
FW      4120
MON     0
Checksum        ▒

sometimes followed by one if these:

H1      0
H2      0
H3      0
H4      0
H5      0
H6      0
H7      0
H8      0
H9      ---
H10     0
H11     0
H12     0
H17     0
H18     0
Checksum        2

the code can be found here https://github.com/kscholty/SmartShuntINA226/blob/2a21a026677e3e6c7b93cf56faf83e25c6283545/src/victronHandling.cpp#L190

as i said, i don't know if its really what you asked for 😄

but, i hope it is. as i definitly want to use a DIY SmartShunt later, when i start using a battery 😃

edit: as for the DIY project, please notice:

Victron VE.direct Text and Hex protocols in order to function as a Battery Monitor. The Hex protocol is only implemented as far as it's required for startup. Furthermore, some fields of the Text protocol are not yet filled correctly.

[helgeerbe]

@madmartin you can download the VE.Direct Protocoll from victron. There you will find all the info which is needed to extend the FrameHandler.

[swingstate]

@swingstate can you record a sample of the text mode output of a victron smartshunt and copy it here? I plan to implement that.

@madmartin I need to wire the shunt to the on-battery ESP and read the data from the debug console, am I right?

[swingstate]

any chance this will be finished?

[bra1nbuster]

I would love to see this feature too, since I currently have to use an additional ESP to get the data from the Victron SmartShunt.

[swingstate]

@bra1nbuster where do you integrate the data? I am using EShome / Homeassistant to trigger automations based on the shunt data. A direct integration into OpenDTU obviously would make more sense in this context; primarily to become independent from Homeassistant.

[schlimmchen]

Interesting topic. Notice that part of the work that is required for this feature is covered by #319 (selection of battery interface in code and web application in particular).

These things are too expensive to buy for fun. So I will not be able to help unless someone donates a smart shunt. Also... time is an issue.

@bra1nbuster and @swingstate Do you have any programming skills? Can't you to it yourselves with a little bit of help? I would be able to give a couple of hints/instructions, and you do the iterative testing against the actual smart shunt?

The Shunt should use dedicated GPIO Pins (needs 2) to interface with the ESP board. [...] hence the "PIN Strategy" should be set up carefully.

I disagree. There is no data output from the ESP to the Victron device at all. As far as I can see, we can and should remove the TX pin from the interface. OpenDTU-OnBattery only listens for messages from Victron devices, which are announced periodically without any interaction. Again, as far as I can see.

[bra1nbuster]

@bra1nbuster where do you integrate the data? I am using EShome / Homeassistant to trigger automations based on the shunt data. A direct integration into OpenDTU obviously would make more sense in this context; primarily to become independent from Homeassistant.

I do the same. I'm using this project ( https://github.com/KinDR007/VictronSmartShunt-ESPHOME ) to get information about my PowerQueen battery into HomeAssistant for monitoring and automations.

@schlimmchen Unfortunately I don't have the right coding skills (C) for this, but I could offer some debug outputs, if needed.

[swingstate]

@bra1nbuster and @swingstate Do you have any programming skills? Can't you to it yourselves with a little bit of help? I would be able to give a couple of hints/instructions, and you do the iterative testing against the actual smart shunt?

@schlimmchen I am not able to code in C, however can do the testing with my main DTU or a similar second device in my new environment with 2x 25,6V 200Ah batteries which will be charged with a 150/35. If you could imagine buying a shunt for the development I would consider to support by sponsoring 50€ - you could sell the shunt once we have the solution implemented.

I disagree. There is no data output from the ESP to the Victron device at all. As far as I can see, we can and should remove the TX pin from the interface. OpenDTU-OnBattery only listens for messages from Victron devices, which are announced periodically without any interaction. Again, as far as I can see.

Agree, no need for the TX since it'S read only.

@bra1nbuster I am using the same project. Works flawless so far. I am wondering if the data from the VE.Direct could be read from two ESP at the same time, basically by using a Y-cable. That way the ESPHome ESP device and a DTU could read the same data from one Shunt. The ESPHome is also only connected to TX & GND on the shunt.

[schlimmchen]

I am wondering if the data from the VE.Direct could be read from two ESP at the same time, basically by using a Y-cable.

The proper solution would be to use RS485 converters and put these devices on an actual bus. Having said that, I don't see any reason why this shouldn't work with UARTs. You still need to convert the power levels. And additional pull-ups on the ESP's RX lines may become an issue at some point, but probably not if only two are used. Once there are two data sources involves, this doesn't work, of course. One data producer, multiple data consumers, that should be fine.

[swingstate]

That's seems to very complicated. I guess then it might be easier to have OpenDTU reading the voltage through the MPPT (which is getting the battery voltage + power aka "Spannung & Strom") via Victon Bluetooth from the Shunt. And, secondly, having the Shunt sending the calculated SOC via ESPhome to Homeassistant.

@helgeerbe Could we implement that DTU can read the SOC via MQTT? Homeassistant could transmit it after ESPhome send an update. That way we would get around changes to the Ve.Direct code stack, until you find time to implement - assuming the MQTT implementation is easier.

[schlimmchen]

Could we implement that DTU can read the SOC via MQTT?

@madmartin works on that (currently waiting for me to continue with #319).

[philippsandhaus]

I am also very interested as I am owning a Victron SmartShunt and it would great to use the State of Charge of the SmartShunt for DPL. I am no C expert, but willing to learn. So far I have managed to extend the VeDirectFrameHandler to handle SOC and some other SmartShunt specific readings and show them in the Web-Interface. But what is missing I guess I would need help with is

1. An approach how to handle two or maybe more Ve.direct devices. At least we would need two (MPPT and SmartShunt). I guess this would mean the possibility to have multiple instances of a VeDirectFrameHandler with different GPIO settings. AM I right?
2. Possibility to select the Smart-Shunt as alternative battery device. I guess #319 would be the basis for this?

[schlimmchen]

So far I have managed to extend the VeDirectFrameHandler to handle SOC and some other SmartShunt specific readings and show them in the Web-Interface.

Nice! That's good progress!

I guess this would mean the possibility to have multiple instances of a VeDirectFrameHandler with different GPIO settings. AM I right?

Yes. I am afraid that this is the hard part. Tackling this should achieve support for (1) a VE.Direct MPPT device, (2) a VE.Direct SmartShunt device, (3) both, (4) any combination of multiple supported VE.Direct devices. For starters, let's focus on (1) and (2) only, i.e., the current support for one MPPT controller should still work well after the change, and additionally, one should be able to connect a single smart shunt.

Do you also own a MPPT device?

I propose the following:

1. The existing VeDirectFrameHandler class becomes the new base class for both devices. Its constructor stays the same, the init() function is fine as well.
2. Create a new VeDirectMpptController class that derives from VeDirectFrameHandler.
3. Identify the parts that are specific to the MPPT support. Move those to VeDirectMpptController. That includes veStruct, which should be renamed to reflect it being specific to MPPT controllers. rxData() stays in the base class. textRxEvent() needs to move to the derived class and will duplicate a lot of code. Let's clean that up at a later point.
4. Make the global Variable VeDirect an instance of VeDirectMpptController. Make sure the MPPT VE.Direct works the same as before the refactoring.
5. Implement SmartShunt-specific code in a new class VeDirectSmartShuntController that also derives from VeDirectFrameHandler (copy VeDirectMpptController and adjust).
6. Dumping all values into a struct (veStruct) is a problem. For now, please copy it, remove MPPT-specific values and add SmartShunt-specific values, and call is SmartShuntStruct or something.

Not all details are covered with these points. In particular, we will run out of HardwareSerial instances, and we already discussed moving to SoftwareSerial instead. You are fine by using HardwareSerial(1) and HardwareSerial(2) for now.

Possibility to select the Smart-Shunt as alternative battery device.

Yes, that PR will add a lot of boilerplate and it will become quite easy to add the SmartShunt as another Battery interface. Since you are the second person to wait for that PR, I will work hard to bring it into a state that allows merging. That will mean that even more features are coming at a later point in time, but that's fine.

That might have been a lot to process. And you might realize that some C++ skill is required as well.

Thanks for helping with this feature! Please don't hesitate to ask for assistance. If you would like code review I am happy to help with that as well.

[philippsandhaus]

Wow, thank you very much for the help and the very detailed proposal! That will help a lot. It will however take a bit for me to set everything up, I guess. Also will also have to build a prototyping board besides my production installation with two ve.direct interfaces. Yes, I also have a MPPT 100/20. So I will be able to test a two-device setup.

Tackling this should achieve support for (1) a VE.Direct MPPT device, (2) a VE.Direct SmartShunt device, (3) both, (4) any combination of multiple supported VE.Direct devices. For starters, let's focus on (1) and (2) only, i.e., the current support for one MPPT controller should still work well after the change, and additionally, one should be able to connect a single smart shunt.

I also think that having one MPPT and optionally one SmartShunt would be a typical setup. I would be very happy if this works and (4) can be done later.

I will come back when there is significant progress or will need further help.

[spcqike]

(4) any combination of multiple supported VE.Direct devices

imo as for now, we only can utilize 1 inverter, we dont need two smartshunts, do we? a second shunt only makes sense for a second battery, but every inverter should only have one battery (even though this battery can consist of multiple batteries, they should all dis-/charge at once through one shunt) (and one could use one battery for multiple inverters)

to make the integration easier, it is possible to "name" (or number) multiple victron devices in the device profile?

right now, victron is integrated like

"victron": { "rx": 22, "tx": 21 },

(TX isn't needed, is it?)

would it be possible to do it like

"victron": { "rx": 22, "nr": 1 }, "victron": { "rx": 23, "nr": 2 },

or

"victron_1": { "rx": 22 }, "victron_2": { "rx": 23 },

[philippsandhaus]

You are right. Multiple shunts do not really make sense and multiple MPPTs will also rarely be used (although it would work perfectly). What I have done so far is therefore introducing a second device type in the device profile section for a shunt and leaving the old one as it is for now to make updates easier. I also left the tx port, but I also guess this could be left out as it not being used in any scenario I can imagine.

I would not go with multiple victron sections and numbering them as it would be better to have an indication if you have a MPPT loader or a shunt.

"victron": { "rx": 22, "tx": 23 }, "victron-shunt": { "rx": 12, "tx": 13 },

[spcqike]

and multiple MPPTs will also rarely be used

afaik thats one frequently asked feature, as several people use 2 Smartsolar Chargers (like two small ones with 2 panels each and/ or a setup in east/west orientation), there are some discussions and questions about this. you allready can create a "single" victron charger (connect multiple smartsolar chargers via bluetooth, so that they charge one battery together), but you only see solar production of the wired charger. so solar passtrough isn't working very well in this setup. (as you don't know how much solar power you have on the second charger)

I would not go with multiple victron sections and numbering them as it would be better to have an indication if you have a MPPT loader or a shunt.

i would only kind of number multiple chargers (if multiple chargers are going to be supported, which imo makes perfect sense). of course one would need a seperate name/device for a shunt, i guess. (or are chargers and shunts distinguishable by some serial output they generate? like different serial number naming/range or even a dedicated string they provide?)

[philippsandhaus]

Your are right, multiple MPPTs are a thing, but I wanted to keep it simple for now and only deal with the shunt first. Btw, I would love to also have a more general solution of multiple chargers. For example I only have part of my solar panels near my battery and my Victron MPPT and have an additional inverter with panels directly connected to my AC. Thus, I would like to additionally charge my battery with an AC-charger, when I producing more than I am consuming, I by controlling it via a Shelly Relais.

of course one would need a seperate name/device for a shunt, i guess. (or are chargers and shunts distinguishable by some serial output they generate? like different serial number naming/range or even a dedicated string they provide?)

You could detect, if you are dealing with a MPPT or a Shunt py checking with two whitelists. But this would implicitly allow for multiple Shunts. I think it easier and more predictive to go with separate sections for shunt and (multiple) MPPTs. But as I said: To keep things simple for now, I would start with one MPPT and one Shunt.

[schlimmchen]

Regarding the pin_mapping.json: There are battery.rx and battery.tx pins already defined. Currently, they are exclusive to the Pylontech battery CAN bus interface. They will be re-used for the JK BMS interface. And you should re-use them once more to support the Victron SmartShunts. For the time being, only one battery interface will be supported. The user will set the serial pins using battery.rx and battery.tx in the pin mapping JSON and then select which data provider shall be used. Based on that setting, an instance of JkBmsController xor PylontechCanReceiver will be created, and any one of them uses the battery.*x pins. If the user selects "Victron SmartShunt", an instance of VeDirectSmartShuntController will be created, which exclusively uses the battery.*x pins (or only the RX pin...).

When @madmartin gets around to implement #293, the user will also be able to select "MQTT" as the battery data provider in the web application. In that case, none of the above mentioned classes will be instanciated, but his implementation will be, and it will just ignore the battery.*x pins defined in the pin mapping JSON.

Does that make sense?

Have a look at https://github.com/schlimmchen/OpenDTU-OnBattery/blob/JK-BMS-support/src/Battery.cpp (this link will be broken some time in the future) which should make the concept of "there is only one battery data provider" clear.

[swingstate]

This make a lot of sense, since you keep the interface to the "BMS" clear (just two Pins) and as a second step allow the user to choose his BMS. Really cool and well explained! This will also prevent a mess up and potential problems if too many options for the BMS would be available to be activated at the same time.

I envision this as a simple drop down on the Battery page, where you choose either Pylontech, Victron Shunt, JK BMS, MQTT etc. Only one of these options become active and use the two battery Pins.

[schlimmchen]

Thanks for agreeing.

I envision this as a simple drop down on the Battery page

Well, yeah... That's why I added the screenshot of the Battery interface setting web page as currently implemented in #319. :grin: The drop down you mentioned is labeled "Data Provider".

[swingstate]

didnt look at the screenshot, sorry 😄 I wonder whether it's worth changing the label "data provider" to something more meaningful like "battery interface" or "battery data provider".

[philippsandhaus]

Regarding the pin_mapping.json: There are battery.rx and battery.tx pins already defined.

Good point! I will use them. This will make it much simpler.

[schlimmchen]

@philippsandhaus I would like to let you know that #319 is ready for review/merge and that you should (re)base your work on top of that branch, or wait until it was merged into development and (re)base your work on development afterwards. Let me know if you don't know what I am talking about.

Please use 3 in config.Battery_Provider to identify the SmartShunt as a battery data provider. 2 will be used by #293.

[philippsandhaus]

Thanks for the info. I have just merged the current development branch in to mine and "reserved" 3 for config.Battery_Provider. I am (slowly) making progress. At least I have now two stubs, one for the MPPT Charger and one for the Shunt. https://github.com/philippsandhaus/OpenDTU-OnBattery/tree/victron-smartshunt

[philippsandhaus]

@schlimmchen I did most of the refactoring proposed in https://github.com/helgeerbe/OpenDTU-OnBattery/issues/205#issuecomment-1694709014 (points 1-6). As I said, C++ is quite new to me, that's why it would be great if you could have a look. At least in my setup everything works as before, but I have split the logic in two new classes 'VeDirectMpptController' and 'VeDirectShuntController' (only partly implemented) deriving from 'VeDirectFrameHandler'.

Finalizing 'VeDirectShuntController' is more or less straightforward, but I struggling a bit with implementing your newly introduced abstract class 'BatteryProvider'. Should it be implemented directly in 'VeDirectShuntController' or should I introduce a new "proxy" class 'VictronBattery' which reads data from the global struct in 'VeDirectShuntController'? If I do the first, it would mean a dependency of 'VeDirectShuntController' in the lib directory to BatteryProvider in the src directory which I guess is not desired?

Any hints would be great. I have learnt a lot meanwhile and I guess the rest of the work will be much quicker ;-)

[schlimmchen]

@philippsandhaus That looks very promising!

I have a bunch of questions.... I need to look at this a while longer. I wrote much more stuff in this comment but had to remove most of it because I was completely off the tracks and wrong. So I cannot comment until I have more time to look at it. Unfortunately I cannot tell you when I will find the time... It might need to wait until the end of next week.

Should it be implemented directly in 'VeDirectShuntController' [...] it would mean a dependency of 'VeDirectShuntController' in the lib directory to BatteryProvider in the src directory which I guess is not desired?

Yes. Absolutely right.

should I introduce a new "proxy" class 'VictronBattery' which reads data from the global struct in 'VeDirectShuntController'?

Yes. Define and implement a new class VictronSmartShunt, which implements BatteryProvider and uses VeDirectShuntController from the library. That seems to be the best option. You'll need a class VictronSmartShuntStats : public BatteryStats (in BatteryStats.(h|cpp)) as well, of course.

I hope that helps. Please keep asking if you need any further assistance. I enjoy helping you getting there :blush:

If you could please clean up the indention of the code you added. You are mixing up spaces and tabs in the same file. Please avoid that. Moreover, please make sure not to include commented-out code in the final version of your implementation.

Also, you need to rebase your work on top of helgeerbe/development eventually. It gets a little messy, I am not sure if you are ready for that. Maybe leave it like it is for now, we can address this later.

[philippsandhaus]

Thanks for the feedback. You are right, some code cleanup will be necessary for sure. Meanwhile I have added the missing pieces. Not everything is shown in the UI, mainly Error states, but the main statistics can be monitored. I have not tested it yet, but the State of Charge should also be available for DPL.

[swingstate]

Philipp, many thanks for your time and effort spent on this request! What you have shared so far looks promising.

I wonder if I can/should test the feature to provide feedback since I own a Smartshunt 500A.

[philippsandhaus]

Sure, testing would be great. You can download the current state here: https://github.com/philippsandhaus/OpenDTU-OnBattery/actions/runs/6222155362

Be sure to define the correct Pins for battery.rx and battery.tx for the pins where your SmartShunt is connected to.

[swingstate]

Do you need the tx to send any data to the shunt? I thought to connect the rx and gnd only since my assumption is that this is read only.

[philippsandhaus]

Yes, tx is not used currently, but the interface is set up with this pin to allow for sending as well. I haven't tried, but it should also be possible to set battery_tx to -1. If not, just take another free PIN, you don't have to attach anything to it.

[swingstate]

ok, so to be clear - which of the 4 Ve.Direct wires do I need to connect to the ESP? At the moment I am using this wiring with ESPhome:

https://github.com/KinDR007/VictronMPPT-ESPHOME#jst-ph-jack

Just GND and TX on the Shunt connected to GND and GPIO 07 on the ESP. Different software obviously, but the wiring should be similar?

[philippsandhaus]

If it works with ESPHOME then it should also work with OpenDTU. Just set battery_rx = 7 and battery_tx = -1. If the latter does not work just set it to an arbitrary and unused GPIO-Pin on your ESP. If you want to be 100% safe you should not connect directly but use some kind of level-shifter / isolator to convert between 3.3V and 5V.

[spcqike]

@philippsandhaus Thanks for your work.

As you have insight into the protocol, do you think this DIY smart shunt project is compatible with genuine Victron shunts? https://github.com/kscholty/SmartShuntINA226

Right now, I don't use a battery at all, so I can't support, test, or help, but later I will use one. Since Victron shunts are quite expensive, I would like to test and build my own based on this project. maybe a DIY shunt is an option for other people, too. 😄

[philippsandhaus]

As the project states it is compatible with with the VE.Direct protocol I do not see a reason why it should not work. I had a quick look at the code and it seems as it sends not all fields like the original Victron SmartShunt, especially error cases. But the most important metrics to make it work with OpenDTU, mainly State of Charge, are supported.

[swingstate]

I might try to build one with my limited soldering skills. Or maybe order an assembled PCB through JLCPCB.

[borzel]

@philippsandhaus It works with my setup:

• HM-300 Inverter
• Victron BlueSolar MTTP
• Victron SmartShunt 500

DPL also has the Battery SoC:

[swingstate]

@borzel for documentation purposes - can you share a few photos how you wired the shunt with the ESP?

[philippsandhaus]

@borzel Great to see it work! In fact, I have it currently only running on a test setup with only the SmartShunt connected. Great to see it working with a full setup.

Do you have any wishes or recommendations of which measurements should be visible in the live view?

[swingstate]

Hi Philipp,

I think we should keep the details on the live view page to the minimum (SOC, current voltage, energy & wattage - negative and/or positive).

I would also suggest to add a page to the info menu for the Shunt. The page should only become visible when the shunt is enabled as a source for the SOC.

The page then should list the more comprehensive details, like for example the home assistant integration does.

Here are the details which the ESPhome based integration provides:

Probably not all details make sense or they are only relevant for certain use cases, like the midpoint voltage for a serial configuration. At least this can give you some input.

I also wonder if the ESP can also do some basic calculations, e.g. charged and discharged kWh/Ah today.

As a 3rd step, I am suggesting to allow choosing certain data field which the shunt provide for MQTT provisioning, so other systems can use the data, e.g. SOC.

[schlimmchen]

I think we should keep the details on the live view page to the minimum

I agree. My metric (for the JK BMS) is this: values that are very important or that change frequently go into the live view. Charging cycles and charged/discharged energy is neither.

I would also suggest to add a page to the info menu for the Shunt.

If you do, please don't name it "Shunt". It's the "Battery" details page, so name it "Battery". I would add to it eventually for the JK BMS, so show the less important and less frequently changing info as well.

I also wonder if the ESP can also do some basic calculations, e.g. charged and discharged kWh/Ah today.

Why?

As a 3rd step, I am suggesting to allow choosing certain data field which the shunt provide for MQTT provisioning, so other systems can use the data, e.g. SOC.

He has done this already in VictronSmartShuntStats::mqttPublish(). I really don't think that the user should be able to select data fields. Most of them should be transmitted to the broker.

@philippsandhaus Nice to see that people are already enjoying your work! :muscle: I hope I find time soon to give more feedback in #452

[spcqike]

I also wonder if the ESP can also do some basic calculations, e.g. charged and discharged kWh/Ah today.

Why?

i guess it's intresting to see, how many Wh/Ah where charged and discharged today, like you can see how many Wh your inverter produced today (which should be the same as discharged Wh, as long as you only have 1 battery and 1 inverter). the Shunt only reports a total value.

but, tbh, i would make this calculation in my monitoring dashboard like grafana or whatever. as i already do it for produced Wh.

[philippsandhaus]

I think we should keep the details on the live view page to the minimum (SOC, current voltage, energy & wattage - negative and/or positive).

I would also suggest to add a page to the info menu for the Shunt. The page should only become visible when the shunt is enabled as a source for the SOC.

Makes sense. But I guess that should be done separately. I guess it should also not be specific for the SmartShunt integration, but for batteries in general.

[schlimmchen]

You can also leave this to me in case you are not interested. And it should indeed be a new PR, the one you prepared in #452 is perfectly feature-complete as it is.