search

victronenergy / venus

Victron Energy Unix/Linux OS
https://github.com/victronenergy/venus/wiki
554 stars 69 forks source link

add node-red aka the main Venus OS Large tracking issue #378

Closed mpvader closed 2 years ago

mpvader commented 5 years ago

Todo

Repo for the node-red pallette is here: https://github.com/victronenergy/node-red-contrib-victron

Node-red discussion on Community is here: https://community.victronenergy.com/questions/549/feature-request-scheduled-virtual-switch-for-venus.html.

Download information + user manual for beta version of all this is here: https://www.victronenergy.com/live/venus-os:large

Initial -possible outdated- story:

What problem does this solve?

The value in this is that it enables capable installers that want to automatically control available outputs such as relays, or software set-points such as the ESS Grid set-point, or Max Charge Current.

How to handle support?

The risk is that it creates endless possibilities, and its impossible to support by Victron: our support staff does not contain software developers; nor PLC experts, and also we don't want to start employing them for the support team. This risk is mitigated by declaring that there is no official support for this; only community support on https://community.victronenergy.com.

Details

Only runs on Venus GX and newer devices

More information about available disk space is here.

Concept drawing

image

mpvader commented 5 years ago

First version of specification is available here.

Next step taken by @pkronstrom is to implement two simple nodes; the battery one and the relay one. Once they are OK; lets do the most complicated one: inverter/charger; to validate that the idea also works for the most complex one.

mpvader commented 5 years ago

Summary of decisions made today:

Example of service + path combinations for Battery Monitor node:

For the Battery Monitor node in Node-red:

Example of service + path combinations for Relay node:

For the Relay node in Node-red:

and so forth; note that those paths need to be verified. Use dbus-spy on a working system for that.

Picture provided by Peter:

image

mpvader commented 5 years ago

Some screenshots of functionality as in the branch on the repo now:

There are two nodes:

image

These are the settings of the battery node:

image

And these are the settings of the relay node:

image

mpvader commented 5 years ago

Adding nodejs & node-red to Venus - update

I've worked on this; the result is an image for the Venus GX; with nodejs v8.4.0 and node-red v0.19.15 installed under daemontools; starting up by default. Change in OE are in the meta-victronenergy/matthijs/wip/node-red branch.

Node-red web UI is available on http://[ip-address]:1880/.

Image can be downloaded here. Install using a USB stick or sd-card; docs here; or copy it with scp onto the device, and then use one of the commandline options of check-update.sh (or do it straight via stdin)

Note that it can only be installed on Venus GX-es that have been formatted with a large partition. How-to-do that is here -> chapter IV, repartition the flash.

To add our node-red work, run:

npm install --production --no-optional -g https://github.com/victronenergy/node-red-contrib-victron/archive/feature/multiplebatteries.tar.gz

The --no-optional is there to make it skip installing abstract-socket; which requires compilation. I don't know yet if abstract-socket is required. Its a dependency of dbus-native; Which as far as I remember had no problem in connecting to local dbus in the work done here; so that would imply it works without.

After installing that module, restart the node-red server by running:

svc -t /service/node-red

Looking at the node-red logs can be done by running:

tail -F /log/node-red/current | tai64nlocal

Whats next?

The D-Bus connection doesn't seem to work yet; which is the next thing to do. It should connect through the system d-bus; just like it does in the Venus-signalk-plugin. Not via TCP. I briefly tried making some modifications; but failed and then gave up / had to do something else.

@pkronstrom please take it from here and let me know if I need to change anything.

pkronstrom commented 5 years ago

I did a fresh install on a new Venus GX. I got the node-red working and victron nodes connecting to dbus following Matthijs' instructions from above:

  1. run the installer image and expand the filesystem following the above instructions
  2. download and scp the node-red swu to Venus GX and Installed from the command line scp venus-swu-...-node-red.swu root@<venus-gx-ip>:~/venus.swu ./opt/victronenergy/swupdate-scripts/check-updates.sh -swu ~/venus.swu
  3. install the master branch npm install --production --no-optional -g https://github.com/victronenergy/node-red-contrib-victron/archive/master.tar.gz

At this point, opening the browser at the Venus GX's address http://<venus-gx-ip>:1880 should show you the node-red ui. However, checking from the logs, it seems that the plugin crashes (on a missing environment variable - this is irrelevant in this case, though)

I managed to get the connection working simply by replacing the tcp-connection string in victronclient.js with a null, which causes the dbus-listener.js to default to using dbus.systemBus() a socket connection.

TL:DR; I need to make a small fix to victronclient.js

pkronstrom commented 5 years ago

TL:DR; I need to make a small fix to victronclient.js

I changed the VictronClient to default to systembus socket connection if the environment variable NODE_RED_DBUS_ADDRESS is not set. See this commit for the changes.

mpvader commented 5 years ago

TL:DR; I need to make a small fix to victronclient.js

I changed the VictronClient to default to systembus if the environment variable NODE_RED_DBUS_ADDRESS is not set. See this commit for the changes.

The comments in that commit and code are a bit misleading

It will not perse connect to the systembus; it can also connect to the sessiondbus; depending on another env variable.

The proper term is probably ‘system or session dbus on a unix socket’. Or something like that; see google.

pkronstrom commented 5 years ago

Short toast for immediate feedback:

screenshot 2019-01-29 at 10 15 13

Static text shown below the switch, if the service is enabled:

screenshot 2019-01-28 at 16 14 55
mpvader commented 5 years ago

Test images are available!

Node-red UI is at http://(ip address):1880

See here for how to install on a raspberrypi. For the Venus GX; first follow chapter IV of this issue to repartition it, and then use the normal offline update procedure to install the swu. For Octo GX, no repartitioning is required; just install the beaglebone swu file.

thomas-pfeifer commented 5 years ago

Hi there. I am a new GitHub user, not knowing exactly whether this is the right way to report an issue related to mpvader's message above. However, I have experienced the following after repartitioning the flash and installing venus-swu-beaglebone-20190310172552-v2.30~32-node-red.swu:

  1. the Node-Red switch does not appear in the GUI
  2. the Node-Red UI can be accessed at IP:1880, but shows two "syntax-error-messages" on startup screenshot

Do I have to install venus-nodes manually? Or is there any other advice to get it running? Thx, Thomas

mpvader commented 5 years ago

Hi @thomas-pfeifer, thanks for reporting!

  1. the Node-Red switch does not appear in the GUI

Thats correct; its always on for now.

  1. the Node-Red UI can be accessed at IP:1880, but shows two "syntax-error-messages" on startup Do I have to install venus-nodes manually? Or is there any other advice to get it running?

No, its not necessary to install nodes manually. And actually we have a new version built recently; could you try them? See next comment.

mpvader commented 5 years ago

New test images are available.

Node-red UI is at http://(ip address):1880

And its default on: there is no GUI switch.

See here for how to install on a raspberrypi. For the Venus GX; first follow chapter IV of this issue to repartition it, and then use the normal offline update procedure to install the swu. For Octo GX, no repartitioning is required; just install the beaglebone swu file.

thomas-pfeifer commented 5 years ago

Hi, I have installed the Beaglebone-SWU 2.30-35 from the link above. Unfortunately, I get the same syntax-error messages in Node-Red UI, as before. Victron nodes are not visible in the UI. When disabling / enabling the "node-red-contrib-victron, 1.1.1" nodes under "User Settings" => "Palette", same syntax errors are thrown. Any ideas?

jhofstee commented 5 years ago

@thomas-pfeifer can you check with Firefox or Chrome? Perhaps it is an IE issue..

thomas-pfeifer commented 5 years ago

Yes, the issue was caused by IE. Tried now with Chrome and Opera - both working fine. Thanks a lot!

pkronstrom commented 5 years ago

Indeed the frontend code used some features not supported by IE. This should be fixed in the next version (Support for IE 10+).

SEspe commented 5 years ago

Hi thanks. Is there a "noob - howto" to install this on 2.30~39 in an RPi ?

mpvader commented 5 years ago

Hi thanks. Is there a "noob - howto" to install this on 2.30~39 in an RPi ?

No. But search for venus raspberrypi image install in Google, and for questions see the Modifications space on community.victronenergy.com.

If you have inprovements for the documentation while going through the process, please do improve the docs yourself. All this Victron on rpi is a community effort.

Enjoy!

thomas-pfeifer commented 5 years ago

Hi, I have a nice Node-Red "Energy Manager" running on a Venus GX, using the test image provided by @mpvader on March 22, based on Venus OS v2.30_35. However, I recently experienced a Remote Console via VRM connection issue, which hopefully could be solved by updating the Venus Firmware, as indicated in the description of v2.31 bug fixes. Do you have a Beaglebone-SWU with current Venus OS and Node-Red available? Otherwise, is there a simple walkthrough to install Node-Red on Venus GX after Firmware-Update? Will there be a solution (somwhen in future) to have automatic FW updates on the Venus, with Node-Red and programmed flows on the device remaining unchanged? Thx a lot in advance, Thomas

mpvader commented 5 years ago

Hi @thomas-pfeifer; yes, actually: this IP address verification issue was solved in v2.30.

I’ll see if I can make a new build for you (v2.32-nodered) tonight.

Wrt the updatability: once we make node-red support official, then you can do updates without (always) losing flows. If it will be 100% watertight remains to be seen.

And, how we’ll manage that for the Venus GX remains to be seen as well; due to the fact that we’ve produced many of them with the small version partition layout. There are solutions; but they’ll take (more) time. And thats whats lacking at the moment: time.

I’ll post here when I have updated images.

mpvader commented 5 years ago

New test image available:

Note that I did not really test it (I opened node-red once; and thats it). One major change wrt last version is that this has version 1.2.0 of node-red-contrib-victron. Previous images had version 1.1.1. See its repo (its public on this github) and then the releases link for whats been changed.

And, I've updated the two branched (matthijs/wip/node-red in openembedded-core and meta-victronenergy)

thomas-pfeifer commented 5 years ago

Hi @mpvader, thank you so much for quick support. It worked great! Remote Console via VRM is now working again, and Node-Red seems to be functioning as before. Surprisingly (for me), my deployed Flows and additional Palette entries completely "survived" the FW update (via SD card), while Victron nodes were updated to v1.2.0. Of course, I have too less knowledge to understand all this stuff - just wanted to let you know that it worked perfect.

mpvader commented 5 years ago

Hi @thomas-pfeifer: thats great to hear. Thanks for the testing report; useful - even though you might not understand everything; its the info I was looking for :-).

mpvader commented 4 years ago

An update; I've just now built, and (very briefly) tested v2.42-node-red-1. Meaning that I took the v2.42 version; and added my node-red changes on top.

Details:

Branches:

Images (built with tag v2.42 for all other branches):

See above threads for how to install on a Venus GX; depending on its serial number you might need to repartition it; which takes some extra effort. Not so for Octo GX and Raspberrypi.

Pipeline:

Required rootfs size is 312k:


root@beaglebone:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root               448390    312618    108101  74% /
devtmpfs                247188         4    247184   0% /dev
tmpfs                   255892       208    255684   0% /run
tmpfs                   255892       212    255680   0% /var/volatile
/dev/mmcblk1p5          499656     76692    386268  17% /data
/dev/mmcblk1p6         4374280      8824   4123536   0% /scratch
root@beaglebone:~# cat /opt/victronenergy/version 
v2.42-node-red-1
Victron Energy
20200111091936
mpvader commented 4 years ago

images are available for download now, see previous comment. if you install & use it; please leave a note here with your experience (all went well is also an experience :) ), whishes or comments. Thanks!

osaether commented 4 years ago

I tried it right now on my Venus GX and it works! I have no devices connected at the moment so there is not much to do.

mpvader commented 4 years ago

An update, I've just now built v2.51-node-red-1. The change is that this adds an image for the new Cerbo GX (which is dual core & a lot faster, so better suited than the Venux GX).

No changes were made to node-red itself; its all the same as previous images, but then built on top of v2.51 rather than v2.42.

Images:

Once installed, navigate to http://[ipaddres]:/1880/ to see the files.

Used branches are meta-victronenergy/matthijs/node-red2 & openembedded-code/matthijs/node-red2, and for the rest all the b2.50 branches (ie. start with venus/b2.50 and reconf to redo this on your own).

warwickchapman commented 4 years ago

Trivial but I cannot edit it myself and some might find it confusing: in the issue text, there is a bullet with suggested deleting indicated:

sparkmark commented 4 years ago

Hey guys ... Ran into an issue. I resized my FS using /opt/victronenergy/swupdate-scripts/resize2fs.sh which was fine.

I copied across the latest install for node-red and installed that using /opt/victronenergy/swupdate-scripts/check-updates.sh -swu /home/root/venus-swu-beaglebone-20200307064055-v2.51-node-red-2.swu

All worked fine, and I was able to access node-red.

However after a few minutes, my venus rebooted itself and reverted back to its previous version (being 2.53). Disk space also reverted back to standard.

mpvader commented 4 years ago

Hey @sparkmark: make sure to disable automatic updates. You can do that already before you install the node-red install.

sparkmark commented 4 years ago

Of course ! ... thanks @mpvader

mpvader commented 4 years ago

No problem; welcome!

@warwickchapman : I’ve seen your msg - I’ll sort it. Thanks!

MARKUS-001 commented 4 years ago

Hi Matthijs, I am using node red quite a bit now since you introduced it to me on the 14th March. It’s actually quite amazing what can be achieved with node red. I have installed quite a few new nodes and created customized dash boards. Grid set point, max inverter power variations, weather forecasts, relay 2 switches on algorithms after checking loads, SOC, combined MPPT energy with no developer skills at all. At first I really struggled to get familiar with the function node, most others are fairly simple. So I googled node red examples then cut and pasted until something started working. With the lockdown in place and many hours later i’m able to produce some meaning full results with the function node and some formulas. I did come a cross one problem though, relay 1, I switched it to manual as per the popup message in node red when trying to use it, but it seems it’s dedicated to to the Venus OS and can’t be used in node red. Is this temporary until further development is done on the node red side of things or will relay 1 always be unavailable for use in node red. Do you think the aux relays on the Multi’s and MPPT’s will eventually be programable in node red or will those remain dedicated to those units through assistants. Is there a relay board available that can connect to the ve.bus or ve.can Thanks Markus.

mpvader commented 4 years ago

Hi @MARKUS-001 thats great to hear, thanks for the update.

mpvader commented 4 years ago

That first venus gx relay should simply work. Something for @sbender9 to look into.

Other relays (multis, mppts) no not anytime soon.

Relay boxes, we’re not working on anything. My best bet would be to buy something online, ethernet or usb connected; & control that from within node red.

RobDuthie commented 4 years ago

Hi All Here is open source PCB, Schematic design for a output unit for the raspi units, it could be adapted to suit. The PCB design just needs a couple more ground plains and is done ready for production. Rob D NZ Victron Raspi Hat.pdf

mpvader commented 4 years ago

New images: Venus 2.60~16-large-image1

Hello!

An update; I've just now built, and (very briefly) tested v2.60~16-large-image1. Which includes both signalk and node-red. And, those two features together in one image, is also what is now the plan: there will be the normal images as we've been building for years now. With all normal & supported functionality.

And for optional install there will be the "large images". Which will have signalk & node-red. Switching between them will be easy: a simple switch in the gui. Updating will be easy to, the online updates will work, just like they do now for the normal images.

All credits for the progress go to @sbender9 by the way - he is doing the actual work here; I've merely packed it together & kicked off the build.

Details of the new available v2.60~16 work

This node-red+signalk build is based on v2.60~16, a beta version. To learn about whats in there, read here.

Details:

Branches:

Downloads

Pipeline:

https://github.com/victronenergy/meta-victronenergy/tree/matthijs/large-image

Screenshots:

The new settings:

image

The new, and for now useless, switch in the Online firmware update menu that makes it check for the "large" image:

image

MARKUS-001 commented 4 years ago

Evening All, I can’t seem to install v2.60_16-large-image1.swu Check for firmware on SD/USB : no firmware found. I reformatted usb Drive as well as SD card to FAT, FAT32 exFAT still no firmware found. I copied v2.60_11 and v2.51-node-red-2.swu back onto the same USB drive, Venus detects the firmware on the USB as soon as it’s inserted.

Markus

mpvader commented 4 years ago

New images: Venus 2.60~16-large-image2

Hello Markus and others,

Sorry, there were two issues with the previous files. Now both fixed:

Venus 2.60~16-large-image2:

Downloads:

https://www.dropbox.com/sh/8fhjmcpjhbr1zhx/AADXQCJjD0L0We82zMk9ylVFa?dl=0

Rafl53 commented 4 years ago

Hello, I have successfully installed the 2.60-16 on my BBB. Node red seems to work perfect. The only thing I am missing is: I don’t habe solar power in W in my Smart Solar MPPT chargers. I can chose all other values but there in not the power in Watts.

Is it a mistake or is it misting on purpose.

Many thanks for your help in advance.

Regards

Rainer

mpvader commented 4 years ago

Hi @Rafl53 , thanks. I'm not sure. Its available on Venus OS internally, on the path /Yield/Power. @sbender9 can you have a look?

mpvader commented 4 years ago

ps: note that above is solar power. So thats power going to the battery plus solar power going to the load output.

Rafl53 commented 4 years ago

D597E042-8112-4DA4-A7F5-82413E06C454 47BFE330-02A6-436D-AF75-55BFBEA842BF Can’t find it.

houser42 commented 4 years ago

Hello, Wanted to test this out. Used the online updater and it seemed to install fine. I assume we then need to manually install npm and node.js also? Could you pass on the correct links for the relevant versions please? Many thanks.

sbender9 commented 4 years ago

@houser42 you need to update using the .swu linked above. This is not in the release candidate yet.

houser42 commented 4 years ago

I see. Version numbers were the same so missed that. Up and running. Interesting! This installed npm -v 6.4.1 and node -v v8.15.0 Ok to update to most recent: Node.js Version v12.16.3 and Npm Version v6.14.5 ? Ok to additionally try and install a lightweight NodeJS server via npm and try out?

sbender9 commented 4 years ago

I’m not sure if updating nodejs will even work, but you can give it try.

You should be fine installing other nodejs packages. Though you will lose them when updating the firmware.

houser42 commented 4 years ago

Ok, thanks. Will try. If I had the old partition system on my Venus GX, has that been updated when I ran the installer? I seem to be out of space before I started ;) I would regardless probably need to allow a little more room for this to work.

sbender9 commented 4 years ago

I think you probably need more space on the rootfs, you need to run resize2fs.sh.

Explained here: https://www.victronenergy.com/live/ccgx:root_access

sbender9 commented 4 years ago

You'll need to load another image of the os first to do that.