search

phoddie / node-red-mcu

Node-RED for microcontrollers
120 stars 18 forks source link
ecma-419 embedded iot javascript moddable node-red xs

readme

Node-RED MCU Edition

Copyright 2022-2023, Moddable Tech, Inc. All rights reserved.
Peter Hoddie
Updated March 24, 2023

Introduction

This document introduces an implementation of the Node-RED runtime that runs on resource-constrained microcontrollers (MCUs). Node-RED is a popular visual environment that describes itself as "a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways."

Node-RED is built on Node.js and, consequently, runs where Node.js does: desktop computers and single-board computers like the Raspberry Pi. Because of the dependency on Node.js, Node-RED cannot run where Node cannot, notably the low-cost MCUs found in many IoT products and popular in the maker community.

These MCUs are able to run the same JavaScript language used by Node-RED thanks to the XS JavaScript engine in the Moddable SDK. However, these MCUs have much less RAM, much less CPU power, and an RTOS instead of a Unix-derived OS. As a result, they require a very different implementation. A typical target microcontroller is the ESP32, running FreeRTOS with about 280 KB of free RAM and a 160 MHz CPU clock.

The implementation converts the JSON descriptions output by Node-RED to JavaScript objects that are compatible with MCUs. The implementation uses standard JavaScript running in the Moddable SDK runtime. The ECMA-419 standard, the ECMAScript® Embedded Systems API Specification, is used for I/O such as access to pin hardware and networking.

This effort is intended to evaluate whether it is feasible to support Node-RED on MCUs. To achieve that goal, the focus is on a breadth rather than depth. Node-RED has a wealth of features that will take considerable time to implement well. This proof-of-concept effort has the following characteristics:

Efficiency is essential for resource constrained device, but it is not a priority yet. For the current goal of determining if it is realistic to support Node-RED on MCUs, the projects just need to run, not run optimally.

Method

The conversion from Node-RED JSON objects to JavaScript objects happens in two separate phase. The first phase happens while building a Node-RED MCU project; the second phase, while running the project.

The following sections look at some examples of the transform, starting from the Node-RED configuration of each object.

HTTP Request

Here is a typical HTTP Request node.

Node-RED generates this JSON for the HTTP Request.

{
    "id": "b0bc5df11987f5b1",
    "type": "http request",
    "z": "8f44c46fbe03a48d",
    "name": "JSON Request",
    "method": "GET",
    "ret": "obj",
    "paytoqs": "ignore",
    "url": "",
    "tls": "",
    "persist": false,
    "proxy": "",
    "authType": "",
    "senderr": true,
    "credentials": {},
    "x": 420,
    "y": 420,
    "wires": [
        [
            "601862f19774933c"
        ]
    ]
}

The nodered2mcu tool converts this JSON to two JavaScript calls. The first creates the node and adds it to the flow. The second initializes the node with a subset of the properties of the Node-RED generated JSON:

createNode("http request", "b0bc5df11987f5b1", "JSON Request", flow);

node.onStart({
    method: "GET",
    ret: "obj",
    paytoqs: "ignore",
    url: "",
    tls: "",
    persist: false,
    proxy: "",
    authType: "",
    senderr: true,
    credentials: {},
    wires: [["601862f19774933c"]],
});

This JavaScript is compiled into the Node-RED MCU Edition project and run on the device. The implementation of the HTTPRequestNode uses this JSON to implement the HTTP request. In this case, the URL for the HTTP request is provided by an Inject node. The next section shows the implementation of the inject.

Inject

This Inject node is configured to send a URL to the HTTP Request node that its output is connected to.

Node-RED generates this JSON for the Inject node.

{
    "id": "b0afa70581ce895a",
    "type": "inject",
    "z": "8f44c46fbe03a48d",
    "name": "GET httpbin/json",
    "props": [
        {
            "p": "url",
            "v": "httpbin.org/json",
            "vt": "str"
        }
    ],
    "repeat": "",
    "crontab": "",
    "once": true,
    "onceDelay": "6",
    "topic": "",
    "x": 190,
    "y": 420,
    "wires": [
        [
            "b0bc5df11987f5b1"
        ]
    ]
}

The nodered2mcu tool optimize this JSON considerably by converting the data to two JavaScript functions.

createNode("inject", "b0afa70581ce895a", "GET httpbin/json", flow);

node = nodes.next().value;  // inject - b0afa70581ce895a
node.onStart({
    wires: [["b0bc5df11987f5b1"]],
    trigger: function () {
        const msg = {};
        msg.url = "httpbin.org/json";
        this.send(msg);
    },
    initialize: function () {
        Timer.set(() => this.trigger(), 6000);
    },
});

By converting to JavaScript, nodered2mcu can include JavaScript functions in addition to JSON values. It creates a trigger function to perform the inject operation described by the JSON and an initialize function to set up a timer that invokes trigger after the initial delay. This optimization allows the Inject node to run more quickly than interpreting the JSON configuration on the MCU. It also tends to require less RAM.

Change

Here is a typical Change node.

Node-RED generates this JSON for the HTTP Request.

{
    "id": "734223794d10f7cc",
    "type": "change",
    "z": "8f44c46fbe03a48d",
    "name": "",
    "rules": [
        {
            "t": "move",
            "p": "payload",
            "pt": "msg",
            "to": "DEBUGGER",
            "tot": "msg"
        },
        {
            "t": "set",
            "p": "one",
            "pt": "flow",
            "to": "one one one one",
            "tot": "str"
        }
    ],
    "action": "",
    "property": "",
    "from": "",
    "to": "",
    "reg": false,
    "x": 680,
    "y": 320,
    "wires": [
        [
            "e34b347db64fcdc8"
        ]
    ]
}

The nodered2mcu tool is able to reduce this large block of JSON to a JavaScript function of just a few lines. The onMessage function it generates is the implementation of the Change node's message handler, so there is no additional overhead in processing this node.

createNode("change", "734223794d10f7cc", "", flow);

node.onStart({
    wires: [["e34b347db64fcdc8"]],
    onMessage: function (msg) {
        let temp = msg.payload;
        delete msg.payload
        msg.DEBUGGER = temp;
        this.flow.set("one", "one one one one");
        return msg;
    },
});

Results

This initial effort successfully runs useful, if small, Node-RED projects on MCUs. They run reliably and perform well. They have been tested on ESP32 and ESP8266 MCUs. The ESP8266 is quite constrained, running at 80 MHz with only about 45 KB of RAM. Both are able to run flows that connect physical buttons and LEDs to the cloud using MQTT. The following sections provide details on what has been implemented.

This effort was helped greatly by Node-RED's small, well-designed core architecture. That simplicity minimizes what needed to be implemented to execute the nodes and flows. Node-RED achieves its richness through the addition of nodes on top of its core architecture. This minimal and modular approach is well suited to MCUs which are designed to be capable enough to do useful work, not to be a scalable, general-purpose computing device.

This effort was also made easier by both Node-RED and the Moddable SDK being built on JavaScript engines that provide standard, modern JavaScript (V8 and XS, respectively). This is essential because the Node-RED runtime is itself implemented in JavaScript and makes JavaScript accessible to developers through the Function node. Additionally, the capabilities defined by ECMA-419 are often a direct match for the runtime needs of Node-RED, making implementation of embedded I/O capabilities remarkably straightforward.

Based on these early results, it seems possible to provide a valuable implementation of Node-RED for MCUs. This would make developing software for these devices accessible to more developers thanks to Node-RED's goal of providing "low-code programming." It would also allow developers with Node-RED experience a path to extend their reach to widely-used MCUs.

Ways to Help

Based on this investigation, bringing Node-RED to MCUs seems both possible and desirable. It is not a small effort, however. It will require expertise in embedded software development, ECMA-419, the Moddable SDK, the Node-RED runtime, and developing flows with Node-RED. Here are some ways you might help:

Please open an issue or submit a pull request on this repository on GitHub. You can also reach out to @phoddie and @moddabletech on Twitter or chat in real time on our Gitter page.

Running Flows on an MCU

Node-RED MCU Edition is a Moddable SDK project. It is built and run just like any Moddable SDK project. Flows run on ESP8266, ESP32, and Raspberry Pi Pico MCUs, and in the Moddable SDK simulator on macOS and Linux computers. Node-RED MCU Edition requires the Moddable SDK from August 8, 2022 (or later).

Of course, the Node-RED flows must be added to the project. The JSON version of the flows is stored in the nodes.json source file. There are two part to moving a Node-RED project to the Node-RED MCU Edition project.

The first is exporting the project from Node-RED.

  1. Open the Node-RED project
  2. From the Node-RED menu (top-left corner), select Export
  3. On the Export tab, select "all flows"
  4. Select Clipboard (not Local)
  5. Select JSON (not Export nodes)
  6. Select "Copy to Clipboard"

Warning: Experienced Node-RED users may choose "selected nodes" or "current flow" on the Export tab in Step 3. Often this does work. However, in some cases the flow fails to operate correctly because Node-RED will not export all required global confirmation nodes. For example, "selected nodes" does not export MQTT Broker nodes required by MQTT In and MQTT Out nodes and exporting "current flow" does not export the global dashboard configuration node required by UI nodes.

The JSON version of the flows is now on the clipboard. The second step is adding this JSON to the Moddable SDK project:

  1. Open the flows.json file in the Node-RED MCU Edition project
  2. Paste in the Node-RED JSON data

Build and run the Moddable SDK project as usual for the target device. The flows.json file is transformed by nodered2mcu as part of the build. If an error is detected, such as an unsupported feature, an error message is output and the build stops.

This process is quick and easy for early exploration. Of course, there are many ways it could be streamlined to improve the developer experience.

Note: When building for Moddable Two, the recommended build target is esp32/moddable_two_io, not esp32/moddable_two. The esp32/moddable_two_io target uses the ECMA-419 compatible FocalTouch touch screen driver which avoids I²C conflicts with sensors.

Structure

The Node-RED runtime executes the nodes and flows. This runtime architecture determines how nodes interact with each other. It also is a key factor in how efficiently the execution uses the limited RAM and CPU power available.

This is a summary of what is implemented in the Node-RED for MCUs runtime:

Credentials

Some nodes contain credentials such as a user name and password. Node-RED stores credentials separately from the flows in a file named flows_cred.json. The credentials files is encrypted with a key stored locally. In addition, Node-RED does not include credentials when exporting flows. All of this is done to prevent accidental sharing of credentials.

Unfortunately, this means that credentials are not available to Node-RED MCU Edition when flows are exported. The nodered2mcu tool provides a solution. If a flows_cred_mcu.json file is in the same directory as the flows.json file processed by nodered2mcu, the credentials are merged back into the flows. In the following, the object on the "8b6e5226cefdb00e" property is merged into the node with ID 8b6e5226cefdb00e in flows.json.

{
    "credentials": {
        "8b6e5226cefdb00e": {
            "user": "rw",
            "password": "readwrite"
        }
    }
}

This solution does not provide protection against unintentional sharing. This is an area for further work.

Nodes

This section lists the supported nodes. The implemented features are checked.

Comment

Comment nodes are removed at build-time.

Debug

Function

Function node implements support for calling done() if function's source code does not appear to do so. The check for the presence of node.done() is simpler than full Node-RED. Perhaps in the future Node-RED can export this setting in the flow so nodered2mcu doesn't need to try to duplicate it.

Inject

Link Call

Link In

Link Out

Catch

Status

Complete

Junction

Junction nodes are optimized out by nodered2mcu by replacing each junction with direct wires between its inputs and outputs. Consequently, Junction nodes have a zero-cost at runtime in Node-RED MCU Edition.

MCU Digital In

Implemented with ECMA-419 Digital class.

If the "rpi-gpio in" node from node-red-node-pi-gpio is used in flows, it is translated to a Digital In node.

MCU Digital Out

Implemented with ECMA-419 Digital class.

If the "rpi-gpio out" node from node-red-node-pi-gpio is used in digital output mode in flows, it is translated to a Digital Out node.

MCU PWM Out

Implemented with ECMA-419 PWM Out class.

If the "rpi-gpio out" node from node-red-node-pi-gpio is used in PWM mode in flows, it is translated to a PWM Out node.

DS18B20

Implemented using "rpi-ds18b20" node from node-red-contrib-ds18b20-sensor, with OneWire bus module and DS18X20 temperature sensor module. Uses simulated temperature sensors on platforms without OneWire support.

MCU Neopixels

Implemented using Neopixel driver from Moddable SDK which supports ESP32 family and Raspberry Pi Pico.

The MCU implementation the Neopixel node calls done() after processing each message so that Complete Nodes may be used. This is useful for chaining animations. The full Node-RED Neopixel Node does not call done().

If the pin is left blank, the global instance lights is used if available.

If the rpi-neopixels node is used in flows, it is translated to a Neopixels node.

MCU Analog

Implemented with ECMA-419 Analog class.

MCU Pulse Width

Implemented with ECMA-419 PulseWidth class.

MCU Pulse Count

Implemented with ECMA-419 PulseCount class.

MCU I²C In

Implemented with ECMA-419 I2C class.

MCU I²C Out

Implemented with ECMA-419 I2C class.

MCU Clock

Implemented with ECMA-419 Real-Time Clock class drivers.

MQTT Broker

Implemented using ECMA-419 MQTT Client draft.

MQTT In

MQTT Out

HTTP Request

Implemented using fetch based on ECMA-419 HTTP Client draft.

HTTP In

Note: The full Node-RED listens on port 1880; the Node-RED MCU Edition, on port 80. A way to configure this is likely appropriate. If there is one in full Node-RED perhaps it can be used.

Implemented using HTTPServer based on ECMA-419 HTTP Server draft.

HTTP Response

Implemented using HTTPServer based on ECMA-419 HTTP Server draft.

WebSocket Client

Implemented using HTML5 WebSocket based on ECMA-419 WebSocket Client draft.

WebSocket Listener

Implemented using HTML5 WebSocket extensions in Moddable SDK and HTTPServer based on ECMA-419 HTTP Server draft.

WebSocket In

WebSocket Out

TCP In

Implemented using ECMA-419 TCP and Listener sockets,

TCP Out

Implemented using ECMA-419 TCP and Listener sockets.

TCP Request

Not yet

UDP In

Implemented using UDP I/O class from ECMA-419.

UDP Out

Implemented using UDP I/O class from ECMA-419.

Range

Change

Switch

Filter

Split

The split implementation has some obscure differences in how it triggers Complete nodes from the Node-RED implementation. These differences are noted in issue reports 3982 and 3983. The intent is for the MCU implementation to match Node-RED once the expected behavior is better understood.

JSON

Template

The Template node uses the mustache.js module.

File Write

The File Write node is implemented using the Moddable SDK of integration LittleFS.

File Read

The File Read node is implemented using the Moddable SDK of integration LittleFS.

MCU Sensor

See the MCU Sensor module's documentation for further details.

CSV

Note: The CSV node is the full Node-RED implementation with small changes to reduce its RAM footprint. This is possible by using the Compatibility Node.

Delay

Note: The Delay node is the full Node-RED implementation (with a small, optional change to reduce its RAM footprint). This is possible by using the Compatibility Node.

Trigger

Note: The Trigger node is based on the full Node-RED implementation with a few changes to take advantage of the nodered2mcu preprocessor. This is possible by using the Compatibility Node.

Join

Note: The Join node is based on the full Node-RED implementation with a few changes to take advantage of the nodered2mcu preprocessor. This is possible by using the Compatibility Node.

Sort

Further optimizations should be possible by having nodered2mcu generate a targeted handle for simple cases (sorting property of a single message, no JSONata, etc.).

Note: The Sort node is based on the full Node-RED implementation with a few changes to take advantage of the nodered2mcu preprocessor. This is possible by using the Compatibility Node.

Batch

Note: The Batch node is based on the full Node-RED implementation with some optimizations. This is possible by using the Compatibility Node.

openweathermap

The API key is not exported by the Node-RED editor. Currently it must be entered manually in the source code of the weather node.

Note: The openweathermap node is the full Node-RED implementation with modifications to use fetch and to reduce its RAM footprint. This is possible by using the Compatibility Node.

Compatibility Node

The Compatibility Node runs nodes written for Node-RED. It is able to run the lower-case example from "Creating your first node" without any changes.

The Compatibility Node is tricky for a number of reasons. At this time, it should be considered a proof-of-concept and a foundation for future work.

The Node-RED nodes, including the lower-case example, are written as CommonJS modules. The XS JavaScript engine supports only standard ECMAScript modules (ESM). This leads to some limitations: export.modules can only be set once and any other exported properties are ignored. Because all modules are loaded as standard ECMAScript modules, nodes run by the Compatibility Node run in strict mode.

While a degree of source code compatibility is provided, the Compatibility Node does not attempt to emulate the (substantial) Node.js runtime. Consequently, it only runs nodes compatible with the features available in the Moddable SDK runtime. Nodes must be added to the Node-RED manifest to be included in the build. See the lower-case example in this repository for an example.

Note: The CompatibilityNode is implemented as a subclass of Node, the fundamental node type of the Node-RED MCU Edition runtime. The CompatibilityClass adds features for compatibility that use more memory and CPU power. For efficiency, internal nodes (inject, split, http-request, etc.) are implemented as subclasses of Node.

Future Work

This prototype is a breadth-first effort to implement all the steps required to execute meaningful Node-RED flows on a resource-constrained microcontroller. For compatibility and completeness, a great deal of work remains. That work requires many different kinds of experience and expertise. Evolving this early proof-of-concept to a generally useful implementation will require contributions from many motivated individuals.

The compatibility goal should be to provide the same behaviors as much as possible so that existing Node-RED developers can apply their knowledge and experience to embedded MCUs without encountered confusing and unnecessary differences. The goal is not to provide all the features of Node-RED, as some are impractical or impossible on the target class of devices.

Runtime

Nodes

Possible future work on built-in nodes:

The built-in nodes are useful for compatibility with the standard Node-RED behaviors. Additional nodes should be added to support embedded features. For example, Display node, etc.

Challenging Dependencies

Several nodes use JSONata, a query language for JSON. This looks like a substantial effort to support and is perhaps impractical on a constrained embedded device. (Note: I now have JSONata building and running some simple test cases. The code size and memory footprint are large. Further exploration is needed to evaluate if JSONata is a viable option to enable, but at least it looks possible)

The JSON node has an option to use JSON Schema for validation.

Implementation Notes

Nodes Providing Manifests

A node may include a moddable_manifest property at the root of its exported JSON configuration. The nodered2mcu tool processes the contents of the property as a Moddable SDK manifest. This allows nodes to automatically include modules, data, and configurations. The MCU sensor and clock nodes use this capability, for example, to include the required driver.

{
    "id": "39f01371482a60fb",
    "type": "sensor",
    "z": "fd7d965ef27a87e2",
    "moddable_manifest": {
        "include": [
            "$(MODDABLE)/modules/drivers/sensors/tmp117/manifest.json"
        ]
    },
    ...

Note: Nodes providing manifests should not use relative paths because the root for the relative path is the location of the flows.json file, which is not consistent.

Contexts

Node-RED uses contexts to "to store information that can be shared between different nodes without using the messages that pass through a flow." Contexts are stored in memory by default, but may also be persisted to a file.

Node-RED memory-based contexts are stored in-memory using a JavaScript Map. This gives a consistent behavior with full Node-RED. Memory-based contexts use memory and are not persisted across runs. Memory-based contexts the default in Node-RED. File-based contexts are an alternative that maybe enabled in the Node-RED settings file. Both Node-RED file-based contexts and memory-based contexts may be used in a single project. Node-RED MCU Edition supports only memory and file-based contexts; use of any other generates an error.

Node-RED file contexts are stored using the Preference module of the Moddable SDK. This provides reliable, persistent storage across a variety of MCU families. Data is stored in non-volatile flash memory. Most implementations guarantee that if power is lost while updating the preferences, they will not be corrupted. Because of the resource constraints of microcontrollers, there are come constraints to be aware of:

Despite these warnings, file-based context data works well when used within their constraints: limit the number of piece of context data stored, keep each as small as practical, and update them infrequently.

An alternative to using file-based contexts is to use a File In and File Out nodes to store state. This is more effort and foregoes the convenience of the file-based context's integration with the Change and Inject nodes, and the context APIs provided by the Function node.

Complete and Catch Nodes

The Complete and Catch nodes receive messages from one or more nodes referenced by their scope property. This organization reflects the UI of the Node-RED Editor, where the user configures the scope of Complete and Catch nodes by selecting the nodes they reference. From a runtime perspective, this organization is inefficient as it requires each node receiving a message to check if the node is in-scope for any Complete or Catch nodes.

The full Node-RED runtime optimizes this by building a map at start-up for the Complete and Catch nodes for each node and then consulting that map for each message. In Node-RED MCU Edition, the organization is inverted by nodered2mcu. Each node that is in scope for Complete and Catch nodes, has a list of references to those nodes, similar to the wires node list used for a node's outputs. This simplifies the runtime beyond the full Node-RED runtime. It also eliminates the need to create a closure for each done() for nodes that are not in scope of any Complete or Catch nodes.

The transformation and optimization of the Complete and Catch node structure by nodered2mcu is a bit obscure but the benefits are not: the runtime memory use and load are reduced so flows run more efficiently on constrained hardware.

Status nodes are implemented using the same optimization.

Thank You

This exploration was motivated by an extended conversation with Nick O'Leary who patiently explained Node-RED to me at OpenJS World 2022. That clear and patient discussion gave me grounding to begin this effort.