Project In / Project Out nodes for interproject communication

[ZJvandeWeg]

Epic

No response

Description

As a: Developer

I want to: Use a custom node to send messages to another gateway

So that: it can continue to process the data

Acceptance Criteria

• [ ] Custom node is introduced: Project In / Project Out like Link In/Link Out
• [ ] MQTT is used as transport, but is transparent (No knowledge of topics or the like)
• [ ] Projects of a team can be selected to transport to, no others.

[knolleary]

This is blocked by #464 - and more specifically, the comments on security here

[knolleary]

We can make some progress on this in parallel to #464 - getting the node repository created and the basic structure of the nodes in place.

• Repo: https://github.com/flowforge/flowforge-nr-project-nodes
  • some outstanding tasks to bring this repo up to our required standards: https://github.com/flowforge/handbook/blob/main/development/packaging.md
• npm package @flowforge/project-nodes

The nodes will require some settings to be provided via settings.js to cover the following:

• FlowForge API URL & token - to be able to query the platform to get a list of projects in the same team as this one.
• Broker connection details - url/credentials

Currently, a Project Access Token doesn't have access to the platform API - only the storage API used by the runtime storage plugin. This will require the project to be able to call an API to get a list of projects in the team. Whether that is the existing /api/v1/teams/XYZ/projects api, or one dedicated for projects to use is TBD. But we should take care to only expose the minimum of information needed (project id/name - no other details)

[Steve-Mcl]

Project links design notes

Concepts

The high level concept of project links is inbox/outbox
inbox being what is sent directly to a project (a project/device has its own inbox)
outbox being what a project broadcasts to anyone who cares to listen

Other notes...

• Some nodes will need a project selector that it gets from HTTP API
• To support a tree like "picker" (as per link-in/out) nodes, it would be necessary to populate $hidden topics for discoverability
  • This is potentially negated if we permit free form sub-topic design
• Node will likely need settings entry in settings.js (to point to API atc)
  • If not found, project-link nodes should not be added to palette & raise error in console
• Instance will likely need Env Vars (device ID etc)

Topic structures

Details (From the proposed Subscription ACLs)

Subscription ACLs	Can subscribe to...	Publish ACLs	Can publish to...
Receive messages from other projects	ff/v1/<team>/p/<own-project>/in/+/#	Send messages to other projects	ff/v1/<team>/p/<any-project>/in/+/#
Receive broadcasts from other projects	ff/v1/<team>/p/<any-project>/out/+/#	Send broadcast message	ff/v1/<team>/p/<own-project>/out/+/#

project to project

• ff/v1/<team>/p/<any-project>/in/<level1>/<leveln>
  • <any-project> is the target you want to send to
  • <level1> is equivalent to motor1 in the example motor/1/speed
  • <leveln> is equivalent to 1 and speed in the example motor/1/speed
  • There must be at least a level1 topic, sub topics are optional
• Free form MQTT style topic e.g. power/status, waterflow/status, monitor/motor/1/speed, etc
• NOTE: the user facing topic must NOT start with a / as is appended to the underlying topic structure
• Thought: may be desirable to identify the source-project. This could be in the topic or in the v5 userProperties?
  • use-case: project-DEVICE1 wants to identify which project-instance updated a certain topic!
  • use-case: project-main receives an event from project-v2 and routes/processes it differently to events from project-v1

broadcast

• ff/v1/<team>/p/<own-project>/out/<level1>/<leveln>
  • <own-project> is the ID of this project
  • <level1> is equivalent to motor1 in the example motor/1/speed
  • <leveln> is equivalent to speed in the example motor/1/speed
  • There must be at least a level1 topic, sub topics are optional
• Free form MQTT style topic e.g. power/status, command/stop, etc
• NOTE: the user facing topic must NOT start with a / as is appended to the underlying topic structure

link call

• ff/v1/<team>/p/<project>/res/<level1>/<leveln>
• The link call node will need to subscribe to a responseTopic. This should be identical to the calculated publish topic that the link call node uses except for index 5 of the full topic should be /res/ instead of /in/

Scenario "Wind Farm"

Regarding the initial design work for "Project Link" plugin, it is helpful to me (and hopefully others) to have a human digestible scenario based on the table of topics (https://github.com/flowforge/flowforge/issues/464#issuecomment-1155300019)

Team

A small team of 6 people manage a small wind farm (20 devices).

• 1 Admin - Environment setup / owner
• 2 IS/IT - maintain the a project for supervisory control and data acquisition
• 3 Engineers - maintain the device projects

The team name for this scenario is "wind-farm-1"

Projects

• project-main - supervisory control and data acquisition
• project-v1 - a project for the 10 old v1 hardware devices
• project-v2 - a project for the 10 newer v2 hardware devices

Devices

• 10 devices use Project project-v1 (old v1 hardware)
• 10 devices use Project project-v2 (new v2 installations)

project-main

• Will receive status info from all devices
• Provide a dashboard for realtime status
• Log various status and CBM data to DBs for long term trending and predictive maintenance
• Ability to command individual devices
• Ability to broadcast commands to all devices

project-v1

• This is the flow files for v1 devices (old internal hardware)
• should emit machine status to project-main (current state) using agreed format V1
• should receive and action commands from project-main using agreed format V1

project-v2

• This is the flow files for v2 devices (newer internal hardware)
• should emit machine status to project-main (current state, CBM data, internal device heartbeats etc) using agreed format V2
• should receive and action commands from project-main using agreed format V1

Overview

  graph TD;
    classDef team fill:#555,stroke:#333,stroke-width:2px
    classDef project fill:#aaa,stroke:#333,stroke-width:2px,color:#000
    classDef device fill:#cef,stroke:#123,stroke-width:2px,color:#000

    PM[/project-main/]:::project---PD1[/project-v1/]:::project;
    PM----PD2[/project-v2/]:::project;

    PD1---PD2;
    PD1---Device_1:::device;
    PD1---Device_2:::device;
    PD1---Device_x:::device;
    PD2---Device_11:::device;
    PD2---Device_12:::device;
    PD2---Device_1x:::device;

Project-Link Nodes: design proposal

Figma: https://www.figma.com/file/hr6tk073P2lIDdLdrczMeK/Project-Link-Nodes?node-id=0%3A1

Base principles

• The design pattern will follow suit of the Node-RED link-in/out/call nodes.
  • The project-link-in/out/call nodes transmission should be considered an envelope and the data passed is a complete msg object
  • This can then include a topic/payload and any other properties required
  • To achieve command/response patterns, users can either...
  • use project-link-in/out nodes to respond
  • use project-link-call to get end-to-end feedback
• The complete node, when pointed at a project-link-out node should be able to determine the envelope was successfully "sent"
• The complete node, when pointed at a project-link-call node should be able to determine the envelope was successfully "delivered and returned"??
• The project-link-call will require a timeout property similar to link-call

Project-Link-Out (Project to Project)

UI (User Facing)

A node can target a specific project and send a msg via the "Send message to project" topic structure e.g.

• target: project-main
• topic: motor/1/speed data: {"uom":"rpm", "value":"400"}

  Backend

  the user facing topic motor/1/speed would be ff/v1/wind-farm-1/p/project-main/in/motor/1/speed under the hood.

Project-Link-Out (broadcast)

UI (User Facing)

A node can send a msg on the "Send broadcast" topic structure e.g.

• target: Broadcast message to all projects
• topic: motor/1/speed

  Backend

  The user facing topic motor/1/speed would be ff/v1/wind-farm-1/p/<own-project>/out/motor/1/speed under the hood. Any project wishing to receive a broadcast would subscribe to ff/v1/wind-farm-1/p/<a-project>|+/out/motor/1/speed

Project-Link-Out (return)

The a project-link-out node requires a means of returning a message that originated from a project-link-call To know where to send the msg next, a LIFO stack in the msg will be required

Project-Link-Out Design

Philosophy - Project-Link nodes will transmit a msg just like the Link-in/out nodes. From the above descriptions, we can conclude the Project-Link-Out node requires the following fields...

• Mode:
  • "Send to specified project"
  • "Return to project link call"
• Target (only applicable for mode:send):
  • "Send message to project"
  • Project ([...list_of_projects_in_THIS_team])
  • "Broadcast message to all projects"
• topic: freeform text entry (mandatory, regex check to ensure no leading or trailing slash and does not contain any wildcards or invalid MQTT characters)

Project-Link-In (Project to Project)

UI (User Facing)

The "in" node will receive a msg that is sent from another project when the topic matches a sent message and the message is directed to this project by selecting the source option "Receive messages sent to this project" Accessing data from project-v1 in project-main, the user would

• select the a source option "Receive messages sent to this project"
• enter a topic e.g. motor/1/speed

  Backend

  • Accessing data from project-v1 motor/1/speed at project project-main...
  • The project-main project-link-in node subscribes to the user facing topic motor/1/speed as ff/v1/wind-farm-1/p/project-main/in/motor/1/speed

Project-Link-In (broadcast)

UI (User Facing)

Accessing data from 1 or all projects in the team, the user would select the source option "Listen for broadcast messages from" then select all projects or a specific project and enter a sub-topic e.g. motor/1/speed

Backend

In accessing data from all projects in the team with topic motor/1/speed, the Project-Link-In node subscribes to ff/v1/wind-farm-1/p/+/out/motor/1/speed In accessing data from a specific project in the team with topic motor/1/speed, the Project-Link-In node subscribes to ff/v1/wind-farm-1/p/<project-id>/out/motor/1/speed

*Project-Link-In** Design

From the above, we can conclude the Project-Link-In node requires the following fields...

• Source:
  • option: "Receive messages sent to this project"
  • option: "Listen for broadcast messages from"
    • List of projects ["all projects", ...list_of_projects_in_THIS_team]
      • Selecting a single <project> from the list, this node would be to only receive broadcast transmissions from a single project
      • selecting all would receive broadcast transmissions from a all projects
• topic: freeform text entry (mandatory, regex check to ensure no leading or trailing slash and does not contain any wildcards or invalid MQTT characters)

project-link-call (for command/response pattern)

The project-link-call node functions like the current link-call node in that it expects a project-link-out node to return the msg along with the original msg sub properties identifying the caller. To perform the command/response pattern, the MQTTv5 property responseTopic should be utilised. The project link will only support project-to-project (no broadcast) The project link call will support nested calls

UI (User Facing)

A timeout should be set. Default 30s (open to debate) A target project must be set A topic must be entered.

Backend

• The link-call node will subscribe to a matching topic in the form of ff/v1/wind-farm-1/p/project-main/res/motor/1/speed which matches up with the publish topic ff/v1/wind-farm-1/p/project-main/in/motor/1/speed For supporting nested calls, an event should be placed in a lookup and data added to the msgs projectLink LIFO stack (for routing)

Design

From the above, this proposal suggests the use of MQTT v5 response topic for routing back to the caller.

• Target: [...list_of_projects_in_THIS_team]
• topic: freeform text entry (mandatory, regex check to ensure no leading or trailing slash and does not contain any wildcards or invalid MQTT characters)

Notes

• Should support nested calls
• Should be restricted to single target?
• sub-topic must not contain wildcards?

Additional thoughts / iteration...

• The UI field name sub-topic is open to debate but should try not to call it topic since it is actually a sub-topic or topic extension
• Ability to list projects and Project-link-in nodes in that project for target selection
• Dynamic Link-call capability (in line with current Link-call nodes)
• If a data schema were permitted this could be advertised for simple tree-browse picking of values at the Project-Link-In node
• Consider means of discoverability of data sent (under the hood retained $topics)
  • Using a hidden retained topic, we could provide "last msg" info in the project-link-in so the user can "copy path" without ever deploying (aids development)
• Show link target as clickable ghost node (in line with current link nodes)
  • May require modifications to node-red to provide deep linking capability
• Should MQTT v5 extended features by utilised to provide richer context
  • e.g. using userProperties we could include a specific ENV VARs (like device ID, event time)

[Steve-Mcl]

@knolleary

The nodes will require some settings to be provided via settings.js to cover the following:

• FlowForge API URL & token - to be able to query the platform to get a list of projects in the same team as this one.
• Broker connection details - url/credentials

Something like (UPDATE 2022/07/25)...

flowForge: {
    baseURL: '${settings.storageURL}',  // FORGE_URL
    teamID:  '${settings.teamID}',  // the team hash passed via env var FORGE_TEAM_ID
    projectID: '${settings.projectID}',  // FORGE_PROJECT_ID
    deviceID: '${settings.projectID}',  // present if deployed to a device?
    projectLink: {
        token: '${settings.projectToken}',
        broker: {
            url: xxxx // value computed from platform
            username: xxxx,  // value computed from platform
            password: xxxx,  // value computed from platform
            clientId: null  // value computed from platform? // << generated from username + ":n" if missing
        }
    }

communication path look like...

node-red project link node (UI) → node-red project link node (Runtime endpoint) → platform (GET '/:teamId/projects') → launcher → node-red project link node ?

[sammachin]

Discussion on where these would fit within our Pricing (https://flowforgeworkspace.slack.com/archives/C032Q63FGG1/p1658494235857459)

[nick]
Where do the Project nodes sit on the pricing plan then?
[nick]
I don’t see them listed in the pricing table
[Sam Machin]
:thinking_face: I don’t think we have any capability in the current billing concepts for controlling nodes based on tier,
We could limit their access based on a template? but templates are not associated to projectTypes,
I guess we could include them in the stack when its built? that could then limit them to specfic project types, and would be preferable over having them installed from npm anyway?
I’m just thinking through the scenarios of a user having 2 projects (small and medium) but they can’t communicate between them.
The ideal place from a user perspective is in the team tier I think, then you get it on all the projects that team has,
[nick]
Personally I think the nodes should be a CE feature
[nick]
… if they are a team tier thing, then we can’t ship them until we have team tiers
[nick]
or if it is a CE -vs- EE thing, then we need to add license checks into all the work I’ve done so far
[nick]
and move the code to the EE part of the code base
[Sam Machin]
I can see that trying to retrofit them to billing isn’t ideal, and we don’t have all the capabilities that we would need to do that right now anyway.
I’m just very aware that we’re still giving away all our value add at the lowest tiers which makes it harder to drive the higher revenue points…. 
[Sam Machin]
However from an FF cloud perspective I can see that these nodes make it more useful to have multiple projects (rather than several sets of flows in one) and that in turn would drive revenue
[Sam Machin]
Based on that lets keep these nodes as a CE feature, but it does highlight the importance of getting the billing and licensing frameworks into a ready state before we build more features 

Summary, Inter-project Communications Nodes will be available to all Editions and Tiers.

[ZJvandeWeg]

I'm worried that we're throwing away differentiating features this way. We should be building the Delta between CE and EE. Every release it stays the same is a missed opportunity from the product sales perspective Op 25 jul. 2022 09:57 schreef Sam Machin @.***>:

Discussion on where these would fit within our Pricing (https://flowforgeworkspace.slack.com/archives/C032Q63FGG1/p1658494235857459)

[nick]

Where do the Project nodes sit on the pricing plan then?

[nick]

I don’t see them listed in the pricing table

[Sam Machin]

:thinking_face: I don’t think we have any capability in the current billing concepts for controlling nodes based on tier,

We could limit their access based on a template? but templates are not associated to projectTypes,

I guess we could include them in the stack when its built? that could then limit them to specfic project types, and would be preferable over having them installed from npm anyway?

I’m just thinking through the scenarios of a user having 2 projects (small and medium) but they can’t communicate between them.

The ideal place from a user perspective is in the team tier I think, then you get it on all the projects that team has,

[nick]

Personally I think the nodes should be a CE feature

[nick]

… if they are a team tier thing, then we can’t ship them until we have team tiers

[nick]

or if it is a CE -vs- EE thing, then we need to add license checks into all the work I’ve done so far

[nick]

and move the code to the EE part of the code base

[Sam Machin]

I can see that trying to retrofit them to billing isn’t ideal, and we don’t have all the capabilities that we would need to do that right now anyway.

I’m just very aware that we’re still giving away all our value add at the lowest tiers which makes it harder to drive the higher revenue points….

[Sam Machin]

However from an FF cloud perspective I can see that these nodes make it more useful to have multiple projects (rather than several sets of flows in one) and that in turn would drive revenue

[Sam Machin]

Based on that lets keep these nodes as a CE feature, but it does highlight the importance of getting the billing and licensing frameworks into a ready state before we build more features

Summary, Inter-project Communications Nodes will be available to all Editions and Tiers.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

[knolleary]

We discussed this at length in today's Product Review meeting and explored the available options.

The conclusion was to look at the feasibility of making this an EE-only feature (available to all teams/projects on an EE licensed platform).

I will scope out what it would take to make the product nodes covered by the EE license.

• From a core code point of view there are two things the platform has to provide for the nodes to work:

  1. the ACL that determines who can publish/subscribe to what topic
  2. the admin endpoint to retrieve a sanitised list of projects in the team (only accessible by the launcher access token).

  We can find a way to only enable those things if EE is enabled.

• We will update the stack install docs/scripts to include the nodes when building stacks. They will get installed regardless of CE/EE licensing

  • The nodes will be published to npm under the Apache2 license, but we will not add them to the Node-RED Flow Library (we don't want general Node-RED users to install them directly).
  • On start-up, the nodes already need to check they have been given the platform config via settings.js to enable them. We can add a license check (or similar) at this point, so the nodes will not register themselves if the platform isn't licensed for them.

[Steve-Mcl]

@ZJvandeWeg can this be closed now?