API Endpoint for Bill of Materials

[joepavitt]

First Iteration - Application-scoped

• Add an API endpoint that returns the full list of npm depdendencies for all instances and devices in a given application
• This shouldn't require any new DB work as we already have all of the relevant data stored on an instance-by-instance basis.
• For a given dependency, as a first port of call, we should have:
  • Dependency/Library name
  • Version installed
  • Instance/Device that it is installed on

Follow Up

• Include general npm data around latest packages to flag libraries that can be updated?
• Flagging versions that have security conerns?
• Extend to team-level view?

  
  ### Tasks

• [ ] https://github.com/FlowFuse/flowfuse/issues/4374
• [ ] Infuse NPM versions into BOM to permit update descisions
• [ ] Add NodeJS version of each instance/device (meta data object?)
• [ ] Add Launcher versions for each instance/device (meta data object?)

Feature Availability

Enterprise Only

[Steve-Mcl]

Unfortunately this is not straight forward. Here are the immediate hurdles I see.

1. Instances
   1. do have a list of "Installed Modules" in "Settings" -> "Palette" however they can be entered with the version * meaning they will install whatever the latest version is on the particular npm registry they are using (typically this is npm but we also provide the ability to provide own/private catalogues
   2. If there is no modules specified, current core code revers to using StorageSettings table (which DOES have specific versions) however from what I can make out, this table is legacy? (superseded/overridden by settings.palette.modules?)
2. Devices (application)
   1. do NOT have the palette -> "installed modules" (and no data in DB)
   2. AFAICS, do NOT report the version of Node-RED to FF though it is possible to specify which Node-RED version, it may not be set. Additionally, the running instance may not have been restarted to pick up a new version specified.
   3. If an application device is running a snapshot then it is possible to get the NR version and modules from its snapshot however, it is not mandatory to be running a snapshot and auto snapshots can be turned off. Additionally, if the user installs additional modules or removes any, this is not stored in the platform unless a snapshot taken.

If we wish to be accurate about we are reporting (actual running versions of node-red and modules vs semver versions), we need to get these from the instance/device and probably need to store them in the DB.

to avoid having to update every launcher in the whole platform, one potential avenue is via the "system info" endpoint Node-RED exposes (introduced in v3). It returns (amongst other things) a list of modules and version and the actual running Node-RED version). However, for devices, since it wont be possible to make that inbound request it is looking very likely the only possible avenue is an update to the device-agent (still looking and still thinking)

The alternative (for instances) is to update the launcher /info endpoint and include more than just the node-red version that is currently sends to the platform. That means all instances will need updating to latest launcher version before the DB is fully populated with usable info.

As it stands, without any launcher / device-agent changes, it looks like the best we can do in a first iteration is:

1. Use the list of modules in StorageSettings - these are the values used to build a settings.js. This data includes Node-RED version and specific modules+version. I need to verify when/where these are updated.
2. If a device is has an active snapshot, use its details (fully aware it will contain the likes of "node-red": "latest" and "@flowfuse/nr-project-nodes":">0.5.0" etc as opposed to actual running versions.

Please let me know if I have missed something guys.

PS, I suspect we may need to get our heads together regarding the brest approach and what (if any) ground work MUST be done before implementing this API.

[joepavitt]

Agreed in Eng. Meeting on direction:

• API is not blocked, let's add API to get this data exposed to FE
• First iteration: semver is okay
• Follow on: actual versions deployed
• Follow on: Equivalent for Devices vs Instances

[joepavitt]

Possible data structures with alternative top-levels:

• Dependency / library name
  • Breakdown by versions
    • Then list of instances/devices for each version

Difficulty there is the "version" could be semver or actual deployed version, so instead:

• Instances/Devices
  • List of Dependencies / library names with semver and actual version included

[Steve-Mcl]

Dependency BOM Design proposal

Endpoint

GET /api/v1/applications/{applicationId}/bom

• Description: The endpoint returns the software BOM (Bill of Materials) for the given application. The BOM is a list of all the dependants and their dependencies.
• Path Parameters:
  • applicationId - The ID of the application for which the BOM is requested.
  • `
• Response:
  • 200 OK - The BOM for the application is returned.
  • 400 Bad Request - The request is invalid (e.g. missing required parameters).
  • 401 Unauthorized - The request is not authorized (e.g. missing or invalid authentication).
  • 403 Forbidden - The request is forbidden (e.g. the user does not have permission to access the application).
  • 404 Not Found - The application with the given ID does not exist.
  • 500 Internal Server Error - An error occurred while processing the request.
• Example:

{
    "id": "app-1234",
    "name": "My Application",
    "type": "application",
    "children": [
        {
            "id": "instance-1",
            "name": "instance 1",
            "type": "instance",
            "dependencies": [
                {
                    "name": "node-red",
                    "version": {
                        "semver": "~4.0.0",
                        "installed": "4.0.0"
                    }
                },
                {
                    "name": "@flowfuse/node-red-dashboard",
                    "version": {
                        "semver": "*",
                        "installed": "1.14.0"
                    }
                }
            ],
        },
        {
            "id": "device-1",
            "name": "device 1",
            "type": "instance",
            "dependencies": [
                {
                    "name": "node-red",
                    "version": {
                        "semver": "latest",
                        "installed": "4.0.2"
                    }
                },
                {
                    "name": "@flowfuse/node-red-dashboard",
                    "version": {
                        "semver": "*",
                        "installed": null
                    }
                }
            ]
        }
    ]
}

About the response data

The response data is structured such that it can be formatted into different consolidated views while providing a level of detail and has future extensibility. For example the version information is provided is both the semver (install spec) and the actual version. This allows for easy comparison of the actual version with the semver version. A future iteration we may wish to add installed since a module can be installed but if the instance/device has not been restarted, it will still be running the previously installed version.

Note that the actual version may not be available until future versions of the application/launcher/device-agent are released. In this case, the actual version will be null.

From this data the client can parse it into an overview which consolidates the dependencies across all instances and devices, or it can be parsed into a detailed view which shows the dependencies for each instance and device.

Initially, it is expected any transformation (consolidated views etc) will be done on the client side, but there is scope via query parameters to allow the server to transform the data into different views.

Another deliberate choice was to provide flat array of objects mixing devices and instances (identified by a type property). This was to simplify iteration and avoid a future where we rename "instances" or add a new type of "device".

Transformations ideas/proposed layout

Example transformation to a consolidated view of semver dependencies:

{
    "applicationId": "app-1234",
    "name": "My Application",
    "report": "semver",
    "dependencies": [
        {
            "name": "node-red",
            "version": "~4.0.0",
            "dependants": [
                {
                    "id": "instance-1",
                    "name": "instance 1",
                    "type": "instance"
                }
            ]
        },
        {
            "name": "node-red",
            "version": "latest",
            "dependants": [
                {
                    "id": "device-1",
                    "name": "device 1",
                    "type": "device"
                }
            ]
        },
        {
            "name": "@flowfuse/node-red-dashboard",
            "version": "*",
            "dependants": [
                {
                    "id": "instance-1",
                    "name": "instance 1",
                    "type": "instance"
                },
                {
                    "id": "device-1",
                    "name": "device 1",
                    "type": "device"
                }
            ]
        }
    ]
}

Example transformation to a consolidated view of modules and installed versions:

{
    "applicationId": "app-1234",
    "name": "My Application",
    "report": "actual",
    "dependencies": [
        {
            "name": "node-red",
            "version": "4.0.2",
            "dependants": [
                {
                    "id": "instance-1",
                    "name": "instance 1",
                    "type": "instance"
                },
                {
                    "id": "device-1",
                    "name": "device 1",
                    "type": "device"
                }
            ]
        },
        {
            "name": "@flowfuse/node-red-dashboard",
            "version": "1.13.0",
            "dependants": [
                {
                    "id": "instance-1",
                    "name": "instance 1",
                    "type": "instance"
                },
                {
                    "id": "device-1",
                    "name": "device 1",
                    "type": "device"
                }
            ]
        }
    ]
}

Example transformation to a consolidated view of modules (actual)

{
    "applicationId": "app-1234",
    "name": "My Application",
    "report": "modules",
    "modules": [
        {
            "name": "node-red",
            "versions": ["4.0.0", "4.0.2"],
            "dependants": [
                {
                    "id": "instance-1",
                    "name": "instance 1",
                    "type": "instance"
                },
                {
                    "id": "device-1",
                    "name": "device 1",
                    "type": "device"
                }
            ]
        }
    ]
}

[joepavitt]

Rather than actual, installed make more sense?

[Steve-Mcl]

Rather than actual, installed make more sense?

Agree. In first iteration, the data we do have (instances) it is actually the "installed" version. Ta.

updated above post

[knolleary]

Looks good to me

[Steve-Mcl]

@cstns not sure if Nick raised a follow up for you or informed you already but we will now start getting installed node-red version populating in the Projects model.

4526

Gimme a shout Monday morning - I wish to not only get this into BOM but also look at feasibility of squaring up by renaming installed and semver to use the same format as Nick did (same as the output from npm outdated e.g. current, wanted, latest) (so long as it's just a ~20m refactoring job, I feel it's worth doing for future cognition)

[cstns]

The frontend changes to update the api to conform to the new keys are minimal, once you update the BOM feature to use the new schema ill create a follow up for the frontend. We can release them sequentially as the feature is allready in prod but the feature flag is disabled.

One thing that catches my eye is the new latest key on the versions payload. Currently the frontend is making external api calls to npmjs to get this info (alongside last updated) which we display in the ui. After we get that sorted, the changes shouldn't take more than 10-15 minutes.

[Steve-Mcl]

One thing that catches my eye is the new latest key on the versions payload. Currently the frontend is making external api calls to npmjs to get this info (alongside last updated) which we display in the ui.

@cstns I know we discussed this, but for anyone following along: The Project model versions will only populate current at this time. The versions properties are objects conforming to the npm outdated output for future scope.

[joepavitt]

Considering this work completed, and have moved the outstanding task to the grandparent Epic: https://github.com/FlowFuse/flowfuse/issues/3694