Device Tagging

[MarianRaphael]

Description

I suggest we consider implementing a "Device Tagging system" as an alternative to Device Groups. The idea stems from the discussions and Steves comment in the Device Group's issue. Device Tagging would offer more flexibility and scalability for users to manage their devices more effectively.

The primary reasons for considering Device Tagging over Device Groups are:

• Flexibility: Tags can be applied and removed with ease, offering more flexible device management.

• Scalability: As opposed to a rigid group structure, tagging could be more scalable, allowing for complex hierarchical relationships.

• Future-proof: Since tags can have properties, they would make future feature enhancements easier to implement with fewer database changes.

Tier

Teams Tier (ee)

Questions & Points for Discussion:

• What would be the conflicts if a device inherits multiple tags?
• How will the UI changes be integrated with the existing navigation layout?
• Would tags be owned by an Application or at the Team level?

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

### Sub / Follow-up tasks
- [ ] https://github.com/FlowFuse/flowfuse/issues/2988
- [ ] https://github.com/FlowFuse/flowfuse/issues/2989
- [ ] https://github.com/FlowFuse/flowfuse/issues/3142

### MVP
- [x] Databse model
- [ ] https://github.com/FlowFuse/flowfuse/issues/2997

[MarianRaphael]

@joepavitt could you add the figma visualization you created?

[MarianRaphael]

For the first iteration, keep it as simple as possible:

• Only one tag per device
• Tags are managed at the team level (similar to devices overview)
• For the UI, see Figma

A nice-to-have feature for the first iteration would be the ability to assign all devices with the same tag to an instance or application at once.

[joepavitt]

Only one tag per device

Architecturally, can't see there really being any more complexities for having multiple tags per device.

[Steve-Mcl]

Only one tag per device

Architecturally, can't see there really being any more complexities for having multiple tags per device.

I agree we should permit multiple tags per device (and also make considerations for _other_futuremodel as a possible future of tagging)

However, there is additional complexity due to the many-to-many relationships necessary to implement multiple tags.

A quick detour / some background

As it stands, we have no means of providing settings for a device assigned to an application. We have no means of providing flexible targeting of settings, nodes, NR version, sections of Env Vars etc

It is my strong desire to design the tagging in a way that permits us (at some later stage) to empower the users with these capabilities. consider this snippet from my current design note:

In a later iteration, Tags should have meaning to the system - e.g.

• a tag may be used to indicate what version of NR the device should be running.
• a tag may be used to provide a selection of env vars to a device.
• a tag may be used to indicate "nodes" a device should install.

When tags support this functionality, the user gets much finer control over the configuration of devices with minimal effort to us (no duplication of settings pages - the tag setup page takes care of that) and minimal effort to the users (by combining tags to make unique and flexible device configurations)

The proposed workflow for this is:

• User creates some tags
  • "NR V3.0.2" // a tag for telling the device agent to install version 3.0.2 of NR
  • "NR V3.1.0" // a tag for telling the device agent to install version 3.1.0 of NR
  • "License Key 1" // a tag that includes an env var "license-key" with a value of "1234"
  • "MQTT Server 1" // a tag that includes an env var "mqtt-server" with a value of "mqtt://mqtt.example.com"
  • "MQTT Server 2" // a tag that includes an env var "mqtt-server" with a value of "mqtt://mqtt.other-server.com"
  • "Modbus Nodes" // a tag that instructs the device agent to include node-red-contrib-modbus in the package.json
  • "OPC UA Nodes" // a tag that instructs the device agent to include node-red-contrib-opcua in the package.json
• The user adds the tags "NR V3.0.2", "License Key 1", "MQTT Server 1" and "OPC UA Nodes"
  • This device will then be automatically configured with the version 3.0.2 of NR, "license-key" and "mqtt-server" env vars, and the OPC UA nodes to be installed.
• The user then creates a device, and tags it with "NR V3.1.0", "Modbus Nodes" and "MQTT Server 2"
  • This device will then be automatically configured with the version 3.1.0 of NR, "mqtt-server" env var, and the Modbus nodes to be installed.

Back to the matter at hand

This is my current design DB schema (WIP):

In all my design iterations, I could see no simple way of supporting multiple tags and a possible future of tags having properties if we did not consider them as being future possibilities at the beginning.

For reference:

I was asked to consider tags as being something as simple as a table of "tag names" for the first iteration however, as far as I can see, this would create difficulties in expanding to both multiple tags and tags with properties - in essence, it would generate a whole world of pain (and difficult database migrations (think SQLite)) if not thoughtfully considered up front.

I am still working on the design and may be swayed away from the many-to-many approach by using a tags JSON field in the devices table (though I like the design above as it avoids modifying the devices table at all)

Which ever way I cut it though, I think there is a need to have 2 tables.

• TagTypes where a tags capabilities are defined - this may be virtual (internal / not a database table)
• Tags where tags are added and configured by a user.

I propose that in the initial design the TagTypes table will simply be something hard-coded in the FF app. It will comprise of a single entry "Device-group" and it will have an empty schema - meaning it is "just a name" for grouping purposes as it has no settable properties.

This simplifies the first iteration to introducing the Tags and DeviceTags tables which can later be augmented with a TagType table and any other future tagging we care to implement.

Potential alternative / simplified DB schema

Please share your thoughts.

@MarianRaphael please link to the "UI figma" you mention in your earlier comment.

[knolleary]

Having reviewed the design and partial implementation, there is a lot of work remaining to get this into a usable state.

I'm currently working out how to strip it back even further so we can have something in place this week - but time is definitely against us.

[knolleary]

For MVP, we are only talking about being able to tag devices with a device-group type of tag. From a UX perspective, I think it will be confusing to introduce the all-singing tagging model when it is only being used for logically grouping devices today. So I think that whilst under the covers it may be tags, the UX should present this as adding the device to a group.

If we want to be able to assign all devices with a particular tag to an instance/application/pipeline stage, that implies once a device is tagged, we have to disable the ability to modify the device individually. Or... does modifying the instance/application of a device cause it to be untagged?

[MarianRaphael]

If we want to be able to assign all devices with a particular tag to an instance/application/pipeline stage, that implies once a device is tagged, we have to disable the ability to modify the device individually. Or... does modifying the instance/application of a device cause it to be untagged?

Building upon https://github.com/FlowFuse/flowfuse/issues/2655#issuecomment-1754689347 to keep it as simple as possible, it's absolutely fine to allow a tag to be assigned only if the device is not bound to another entity like an instance or application. After the tag is assigned, an additional individual assignment wouldn't be possible.

[joepavitt]

From a UX perspective, I think it will be confusing to introduce the all-singing tagging model when it is only being used for logically grouping devices today.

I agree

[joepavitt]

I am hoping that we steer towards the removal of Devices being assigned to an Instance, it causes confusion on too many levels, and all functionality can be achieved by Devices at the Application-level once we have Device Tags in, and have them assignable in a Pipeline Stage.

The main complexity to tackle at the Device Tag/Group level would be definition of Device settings (i.e. everything the Device currently gets from Instance settings/template)

[joepavitt]

it's absolutely fine to allow a tag to be assigned only if the device is not bound to another entity like an instance or application

Devices belonging to an Application absolutely should have tags. Introducing tags/groups are for this exact purpose. Applications are used for organise the ecosystem of Devices/Instances.

To push flows to a Device, it'll need to be in an Application anyway, as it needs to be part of the Application's Pipeline. This is also why I've been trying to advocate for "Application" being a dropdown option when you first create a Device. Devices have no reason to be Unassigned" by default.

[MarianRaphael]

I agree, but this is not what I have written: the process of assignment is relevant due to potential conflicts.

Assignment of Tags

Device A is currently connected to Instance A with Target Snapshot X set. Device B is currently connected to Application B and is part of a Pipeline (https://github.com/FlowFuse/flowfuse/issues/2756).

Scenario A - No Tag limitation during assignment: Device A and B receive the same Tag, and the Tag is assigned to a Pipeline in Application C with Target Snapshot Y. If the Pipeline is executed, we have a conflict for Device A; we don't know if X or Y is set as the target.

Scenario B - Tag limitation for Devices bound to an Instance: Device A first has to be disconnected - so there are no conflicts. Device B is now part of two pipelines. The result is no conflict, but we cannot say if all Devices with the same tag are running the same snapshot.

Scenario C - Tag limitation - Assignment only to unbound Devices: Devices with the same tag in a pipeline are running the same snapshot.

Conclusions

Scenarios B and C are functional. In the first iteration, I thought Scenario C would reduce the scope. I am more than open to moving to B directly.

[joepavitt]

Scenario A

Pipeline in Application C with Target Snapshot Y.

There is no concept of a "Target Snapshot" in Applications. That's Instances only, and something I am proposing we remove (eventually)

[joepavitt]

Scenario B - Tag limitation for Devices bound to an Instance: Device A first has to be disconnected - so there are no conflicts. Device B is now part of two pipelines. The result is no conflict, but we cannot say if all Devices with the same tag are running the same snapshot.

Not sure I follow this. Why would Device B be part of 2 x Pipelines?

[joepavitt]

For a Pipeline Stage, you would choose between: Device, Instance, Tag.

If Device - that stage is representative of 1 x Device Only If Instance - that stage is representative of 1 x Instance Only If Tag - that stage represents any Devices assign to that Tag

[MarianRaphael]

Not sure I follow this. Why would Device B be part of 2 x Pipelines?

• Device B is currently connected to Application B and is part of a Pipeline
• Device A and B receive the same Tag
• The Tag is assigned to a Pipeline in Application C
• Device B is now part of two pipelines in Application B and C

[joepavitt]

The Tag is assigned to a Pipeline in Application C

My understanding was that Devices are still Application-bound. A Device Tag in a Pipeline Stage would only retrieve Devices in that Application, with that Tag. So this wouldn't be possible.

How we define Tags, and whether they're Application-bound too is open for Debate though, as can see pros/cons for any of that.

[MarianRaphael]

Devices are still Application-bound. A Device Tag in a Pipeline Stage would only retrieve Devices in that Application, with that Tag.

This would mean that, in addition to the tag, you would need to assign each device individually to an application?

[joepavitt]

you would need to assign each device individually to an application?

Yes, which is what I've been asking for, for some time. Including DMs to you on Slack 😄

[joepavitt]

If Devices can be unbound entirely, free from anything, then we have no purpose for Applications?

[MarianRaphael]

Why not facilitate easier management with tags and assign a tag to an application? All devices with this tag are assigned to the application.

[joepavitt]

Why not facilitate easier management with tags and assign a tag to an application? All devices with this tag are assigned to the application.

You still need to manage the Device's being assigned to a tag. So, being assigned to a tag, vs. being assigned to an Application is no difference?

[MarianRaphael]

So, being assigned to a tag, vs. being assigned to an Application is no difference?

Right, with tags, it's easier to manage large amounts of devices (always referring to the first iteration).

[joepavitt]

I'm trying to ensure consistency with our model by making sure that Instances & Devices are at the same level:

If we're introducing the concept that a Device must be part of a Tag/Group, and then that Tag can be owned by an Application, then the Architecture becomes:

[joepavitt]

We're also in effect then saying that a Device can just be owned by a team, and not by a Tag/Application at all too, which results in:

All feels very messy to me.

Devices/Instances should be treated the same, hierarchically, imo.

[MarianRaphael]

If we're introducing the concept that a Device must be part of a Tag/Group, and then that Tag can be owned by an Application, then the Architecture becomes:

Not "must". Why is this inconsistent? When a tag in this scenario is defined as one or more than one device, it is hierarchically exactly the same as visualization one.

We're also in effect then saying that a Device can just be owned by a team, and not by a Tag/Application at all too, which results in:

No, it's assigned either to Tag and than to Application or directly to application without tag. How does this model come about?

[MarianRaphael]

I'll set up a meeting for Friday

[MarianRaphael]

Device Tagging is replaced by Groups, see detailed issues