Node-RED Node should be packaged properly (and published to npm)

[hardillb]

NOTE issues are for bugs/feature requests not support. Please ask at https://stackoverflow.com/questions/tagged/volttron for help

Description of Issue

example/NodeRED needs a package.json to include the python-shell dependency and could then be published to npmjs.org

{
    "name"         : "volttron",
    "version"      : "0.0.1",
    "description"  : "A Volttron node" ,
    "dependencies": {
        "python-shell": "^1.0.4"
    },
    "keywords": [ "node-red" ],
    "node-red"     : {
        "nodes": {
            "volttron": "volttron.js"
        }
    }
}

Please follow the packaging guide here https://nodered.org/docs/creating-nodes/packaging

Also please clarify what version of python-shell is required.

Affected Version

Screenshots

Expected

Actual

Steps to Reproduce

Additional Details

https://stackoverflow.com/questions/70326753/install-a-node-red-module-from-a-git-repo/

[bbartling]

I also submitted a git issue for the PythonShell. I'm still tinkering around with this getting the same error in Node - PythonShell is not a constructor.

https://github.com/extrabacon/python-shell/issues/261

[bbartling]

I almost think this effort could be abandoned of this older example code of a Node Red connection. I have been tinkering around with David Rakers fork with the REST API features and I think it will suffice anything I would need in Node Red. Also on David's fork of VOLTTRON there is a pubsub end point for the VOLTTRON message bus through web sockets so I don't I think this older method of connecting Node Red to VOLTTRON would be needed..

My 2 cents I think the Node Red Example could be deleted or abandoned. And Node Red can just use the API that David Raker is developing which seems to be working for what I need it for.

[craig8]

Thanks @bbartling would there be any value in having an "integrating with other platforms" type of paper or something with node red?

[bbartling]

absolutely, this maybe a good fit for us on our next round of research funding.

So for my use case in grid interactive efficient buildings (GEB) edge device deployments:

1. I think creating the logic in Node Red to override setpoints on the HVAC system will be much easier than a Python script. Much easier I mean for people like myself with PLC programming background where the look and feel of node red is more what I am used to. Even the BAS industry could possibly pickup VOLTTRON if they can make it work via a GUI with a flow type programming editor like Node Red versus a VOLTTRON "agent" developer, I think a good one would have a computer science theory background and be capable of writing HVAC control agents in a FSM Finite State Machine format which is definitely not the Building Automation Contracting world that doesn't code, they need GUIs.

2. Even I can setup dashboard on an edge device with Node Red with a few widgets and clicks, I think there would use case for this. Not to chart data like a Grafana dashboard in the cloud but have the edge device represent some sort of dashboard locally in the building on a LAN URL for what the current demand response payload is from the utility provider, if demand response is active, and also what equipment is being overridden in the building to shed electrical consumption. (BACnet writes to HVAC, Lighting, BAS points, etc.)

3. I think this opens up the door for more users of VOLTTRON where I do follow the Node Red Forum and there is a large paper trail of BACnet drivers for Node Red are not good, like the javascript BACnet stacks haven't been updated in a few years and they also don't support BACnet MSTP twisted pair networks. (rs 485) I think the home automation Node Red industry is a huge global community and they can get into commercial type work where they could still use what they know (Node Red) but then use the driver framework of VOLTTRON to interact with building automation systems (or whatever) where the VOLTTRON drivers work well. In my opinion the Node Red home automation industry would pickup VOLTTRON much easier than the BAS contractor industry.

White paper sounds great : )

[meeki007]

@bbartling I saw your issue https://github.com/extrabacon/python-shell/issues/261 and followed it here. you need to use a dynamic import you used

var PythonShell = require('python-shell');

try

var {PythonShell} = require('python-shell');

Why am I posting here..... I was starting work to bring python one liners and or scripts for using tensorflow and tensor TF. into a node for node-red for personal use....... I've written a few node-red nodes https://flows.nodered.org/user/meeki007?type=node and thought you might better lay out the requirements you looking for in a node and I could make a one off for everyone to use. I'm thinking something like the exec node but for python.

[bbartling]

@meeki007 thank you

[bbartling]

So trying this all out again with npm install with a package.json

All of the files from the old Node Red example were copied to ~/.node-red/node_modules/volttron

It seems to install OK, assuming this version of python-shell is good:

package.json

{
    "name"         : "volttron",
    "version"      : "0.0.1",
    "description"  : "A Volttron node" ,
    "dependencies": {
        "python-shell": "^1.0.4"
    },
    "keywords": [ "node-red" ],
    "node-red"     : {
        "nodes": {
            "volttron": "volttron.js"
        }
    }
}

AND making 2 changes in the volttron.js file for var {PythonShell} = require('python-shell');

On the Node Red side this crashes when I drag the VOLTTRON object out of the Node Red pallet and connect the debug Node Red block & deploy. On the Node Red side:

26 Dec 13:56:00 - [info] Server now running at http://127.0.0.1:8000/
26 Dec 13:56:00 - [info] Starting flows
26 Dec 13:56:00 - [info] Started flows
26 Dec 13:56:40 - [info] Stopping modified flows
26 Dec 13:56:40 - [info] Stopped modified flows
26 Dec 13:56:40 - [info] Starting modified flows
26 Dec 13:56:40 - [info] Started modified flows
26 Dec 13:56:40 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron
26 Dec 13:56:40 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron/node_red_subscriber.py
26 Dec 13:56:40 - [red] Uncaught Exception:
26 Dec 13:56:40 - Error: spawn /usr/lib/python3.8 EACCES
    at Process.ChildProcess._handle.onexit (internal/child_process.js:268:19)
    at onErrorNT (internal/child_process.js:470:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

It looks like subscriber.py has some issues? This file is in the link above. Thanks for any help

[meeki007]

Looks like a issue with accessing python3

ssh (bring up a terminal) on your device running node-red. Make sure you are able to run python3

if you can type in: $ which python3 what does it output???

note: permission-related issue is most the time when I get a EACCES error. Might try setting permissions for the subscriber.py to that of the same user you used to install node-red.

[bbartling]

I am running this on Ubuntu 20.04, VOLTTRON is running in a virtual environment on 1 tmux pane and I am starting node-red on a different tmux pane on the same SSH session.

In the Node Red tmux session:

ben@ben-HP-ProBook-6550b:~/.node-red/node_modules/volttron$ which python3
/usr/bin/python3

In the VOLTTRON tmux session:

(volttron) ben@ben-HP-ProBook-6550b:~/Desktop/volttron$ which python3
/home/ben/Desktop/volttron/env/bin/python3

In the link to the files that come with VOLTTRON that are to be copied into /.node-red/ directory, the file volttron.js I may have had these wrong:

module.exports = function(RED) {
    // Set these variables to be valid file paths
    var volttron_env = '/home/ben/.volttron';
    var volttron_home = '/home/ben/Desktop/volttron';
    var python_path = '/usr/lib/python3.8';

I changed the var python_path like this below for the VOLTTRON virtual env, is that correct???

module.exports = function(RED) {
    // Set these variables to be valid file paths
    var volttron_env = '/home/ben/.volttron';
    var volttron_home = '/home/ben/Desktop/volttron';
    var python_path = '/home/ben/Desktop/volttron/env/bin/python3';

In terminal starting node-red again that has a saved flow of the VOLTTRON input object in the node-red pallet: Thanks @meeki007 for the tips...

Welcome to Node-RED
===================

27 Dec 07:22:13 - [info] Node-RED version: v2.0.5
27 Dec 07:22:13 - [info] Node.js  version: v12.22.8
27 Dec 07:22:13 - [info] Linux 5.11.0-43-generic x64 LE
27 Dec 07:22:14 - [info] Loading palette nodes
27 Dec 07:22:15 - [info] Dashboard version 2.30.0 started at /ui
27 Dec 07:22:15 - [info] Settings file  : /home/ben/.node-red/settings.js
27 Dec 07:22:15 - [info] Context store  : 'default' [module=memory]
27 Dec 07:22:15 - [info] User directory : /home/ben/.node-red
27 Dec 07:22:15 - [warn] Projects disabled : editorTheme.projects.enabled=false
27 Dec 07:22:15 - [info] Flows file     : /home/ben/.node-red/flows.json
27 Dec 07:22:15 - [info] Server now running at http://127.0.0.1:8000/
27 Dec 07:22:15 - [info] Starting flows
27 Dec 07:22:15 - [info] Started flows
27 Dec 07:22:15 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron
27 Dec 07:22:15 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron/node_red_subscriber.py
27 Dec 07:22:16 - [red] Uncaught Exception:
27 Dec 07:22:16 - SyntaxError: Unexpected token W in JSON at position 0
    at JSON.parse (<anonymous>)
    at PythonShell.asJson (/home/ben/.node-red/node_modules/volttron/node_modules/python-shell/index.js:375:21)
    at /home/ben/.node-red/node_modules/volttron/node_modules/python-shell/index.js:326:42
    at Array.forEach (<anonymous>)
    at PythonShell.receiveInternal (/home/ben/.node-red/node_modules/volttron/node_modules/python-shell/index.js:322:15)
    at PythonShell.receiveStderr (/home/ben/.node-red/node_modules/volttron/node_modules/python-shell/index.js:306:21)
    at Socket.<anonymous> (/home/ben/.node-red/node_modules/volttron/node_modules/python-shell/index.js:109:22)
    at Socket.emit (events.js:314:20)
    at addChunk (_stream_readable.js:297:12)
    at readableAddChunk (_stream_readable.js:268:11)

[bbartling]

For what its worth I created some gists of the files being modified here: ben@ben-HP-ProBook-6550b:~/.node-red/node_modules/volttron$

[bbartling]

All just for fun I updated the python-shell version in the package.json to this:

{
    "name": "volttron",
    "version": "0.0.1",
    "description": "A sample node for node-red",
    "dependencies": {
        "python-shell": "^3.0.1"
    },
    "keywords": [
        "node-red"
    ],
    "node-red": {
        "nodes": {
            "volttron": "volttron.js"
        }
    }
}

Now there is at least no Node Red errors at least npm doesn't crash. : )

In the terminal on the Node Red side I do see some sort of VOLTTRON error:

27 Dec 13:17:35 - [info] Starting flows
27 Dec 13:17:35 - [info] Started flows
27 Dec 13:17:35 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron
27 Dec 13:17:35 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron/node_red_subscriber.py
27 Dec 13:17:48 - [error] [volttron-input:d38957dc576f0b8b]

On the VOLTTRON side I do see this the authentication for the Javascript processes to VOLTTRON, @craig8 does anything look funny to you? Or seem OK on the VOLTTRON side?

2021-12-27 13:17:31,007 (actuatoragent-1.0 52337) __main__ WARNING: Timeout(20.0 seconds)
2021-12-27 13:17:31,007 (actuatoragent-1.0 52337) __main__ DEBUG: sending heartbeat
2021-12-27 13:17:38,115 () volttron.platform.auth INFO: AUTH: After authenticate user id: 'e2e7fbcb-514b-4f4c-8639-d8a7941d9aeb', b'dae9d345-8c65-4609-ac72-78a221887ec6'
2021-12-27 13:17:38,115 () volttron.platform.auth INFO: authentication success: userid=b'dae9d345-8c65-4609-ac72-78a221887ec6' domain='vip', address='127.0.0.1', mechanism='CURVE', credentials=['hLkxOdGwmu_fSoxTQ4qQhFhTtgW-iDmW0F0H20A3pi8'], user='e2e7fbcb-514b-4f4c-8639-d8a7941d9aeb'
2021-12-27 13:17:51,010 (actuatoragent-1.0 52337) __main__ WARNING: Timeout(20.0 seconds)
2021-12-27 13:17:51,010 (actuatoragent-1.0 52337) __main__ DEBUG: sending heartbeat 

Still curious what would cause errors on subscriber.py? Thanks all for the time to read respond keeping this thread alive.

[bbartling]

@craig8 is this subscriber.py like a simplified version of the open ADR agent that uses std.in/out with gevent? Just curious if subscriber.py could be rewritten better?

[shwethanidd]

@bbartling: @bonicim is taking a look at it and will get back to you when he has answers. Thanks!

[meeki007]

Out of necessity I've just started learning some python a few weeks ago. Saying I'm a python noob is a gross overstatement. However I can try my best and sort out the nodeJS/node-red stuff for you.

Note: All file line numbers and reff are from your gists @ https://gist.github.com/bbartling/37608063855be98bf0a035f983d40e3e

the warns from node-red:

27 Dec 13:17:35 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron
27 Dec 13:17:35 - [warn] [volttron-input:d38957dc576f0b8b] /home/ben/.node-red/node_modules/volttron/node_red_subscriber.py

is normal behavior. the creator of volttron.js is using it for communicating to the user. line 29: in volttron.js

node.warn(__dirname);

line 32: volttron.js

node.warn(scriptPath);

So you can ignore that output as it looks correct

the last line (error) from node-red:

17:48 - [error] [volttron-input:d38957dc576f0b8b]

this is coming from line 43 or 71: in volttron.js

if (err) node.error(err);

when its trying to run the node_red_publisher.py its getting an error

as the output of this error is blank in node-red (null/emptyWTF) I can only take a stab at this as my python is lacking. Idea 1 in node_red_publisher.py starting at line 15:

# Log warnings and errors to make the node red log less chatty
utils.setup_logging(level=logging.WARNING)
_log = logging.getLogger(__name__)

where is this log sent to? try rewriting this py to send all the info to node-red and MAKE it chatty :)

Idea 2 I've had this issue once when sending errors to node-red from one of my contrib nodes. The issue was that it was an array or object or somthing ..... cant remember. What I do remember is I used JSON.stringify() to get the dang error into something human readable

try changing ( in volttron.js ) starting at line 42

pyshell.end(function (err) {
                node.error(err);
            });

to

pyshell.end(function (err) {
                 node.error(JSON.stringify(err));
            });

NOTE: I thing this should have been written as a if statement as this might just be a null value or undefined for the err and its why your getting a blank error in node-red........

and

try changing ( in volttron.js ) starting at line 70

PythonShell.run(scriptName, options, function(err, result) {
                    if (err) node.error(err);
                    if (result) node.warn(result);
                });

to this

PythonShell.run(scriptName, options, function(err, result) {
                    if (err) {
                         node.error(JSON.stringify(err));
                    }
                    if (result) node.warn(result);
                });

hope this helps.