search

FlowFuse / flowfuse

Build bespoke, flexible, and resilient manufacturing low-code applications with FlowFuse and Node-RED
https://flowfuse.com
Other
253 stars 60 forks source link

Improve Snapshot lifecycle handling #3622

Open knolleary opened 4 months ago

knolleary commented 4 months ago

Description

We need to improve our general lifecycle handling of snapshots.

We have seen cases where a snapshot gets orphaned (as its Instance/Device is deleted). The snapshot may still be 'in-use' as the target for a device group - however it can no longer be deployed as the Instance/Device no longer exists to retrieve the credentialSecret needed to decrypt the credentials.

Also, once in this state, the snapshot doesn't appear anywhere in the UI because there is no relation between the snapshot and Application/Instance/Device in order to retrieve it.

Problems to solve:

  1. A snapshot needs to be deployable even if its originating device/instance has been deleted
  2. A snapshot needs to be findable in the UI even if its originating device/instance has been deleted
  3. Snapshots live forever - once the device/instance is deleted the snapshots remain and don't get tidied up (even when the team is deleted)
  4. No visibility on what a device-groups current targetSnapshot is
  5. No visibility on what snapshots are currently 'in-use' (ie currently a target snapshot for a device and/or device group)

Solutions:

  1. Migrate the credentialSecret to live with the snapshot. Allows the originating device/instance to be deleted without losing access to the snapshot
  2. Add ApplicationId to snapshot. This will be set based on the originating Instance/Device. Need to consider what happens if a Device is then reassigned to a different Application. Do the snapshots move over as well?
### Tasks
- [ ] https://github.com/FlowFuse/flowfuse/issues/3644
- [ ] Add ApplicationId to snapshot model and keep in sync with owning instance/device
- [ ] https://github.com/FlowFuse/flowfuse/issues/3672
- [ ] Add ability to remove snapshot from a device group
Steve-Mcl commented 2 months ago

Solutions:

  1. Migrate the credentialSecret to live with the snapshot. Allows the originating device/instance to be deleted without losing access to the snapshot
  2. Add ApplicationId to snapshot. This will be set based on the originating Instance/Device. Need to consider what happens if a Device is then reassigned to a different Application. Do the snapshots move over as well?

RE point 1:

I believe this is done right? (snapshot table now has credentialSecret field)

RE point 2: "Do the snapshots move over as well"

This is tricky. A snapshot may also be assigned to a Device Group or another device that will remain in the original application. A question that has been posed before is, when a device or group is assigned a snapshot, should we generate a copy and let the thing own that copy? This would remove this problem (but there will likely be a new set of considerations).

Perhaps a first step would be to have the device abandon its snapshots (clear snapshot.DeviceId) when being removed from an application or maybe when assigned to another application? Consider "dangerous" or sensitive data may be in a devices snapshots, moving those snapshots to another application may not be desirable?

Related to the above, another point of this overall story is to add ApplicationId so that snapshots are never really orphaned, meaning if we do have devices abandon its snapshots, they still exist and are still accessible to be exported/imported.

One hurdle of a snapshot with only ApplicationId is that it wont appear in a device or instance snapshots list & therefore cannot be applied to anything without first exporting/importing. As part of this, or a future task, we might wish to provide snapshot assignment from the application level. Alternatively, we permit Instance snapshot view to see All Application snapshots (like a device can)?

Proposed approach for this part of the story (in order of implementation)

  1. Add ApplicationId to snapshot model
    1. The field should be populated upon migration (set to the ApplicationId of its associated DeviceId/ProjectId)
    2. Consider (in the migration) purging any snapshots that have already been abandoned (since we cannot be certain which application they originally belonged to)?
  2. Have devices abandon their snapshots and ensure they can still be viewed/managed/downloaded at the (original) application level
    1. This should include some simplification of DB queries on the application level views
    2. Have devices abandon their snapshots when moved to another application
      1. These 2 tasks are deliberately pooled into this piece of work since doing either one separately would create either a gap in access to the snapshots or an issue where a device snapshots become available to another application (potentially, need to think it through)

stretch goal

Since the above would provide access to abandoned snapshots, there is a means of downloading orphaned snapshots and uploading to a device/instance, but that is not great UX. The next task could be to:

  1. Add possibility of applying a snapshot with no DeviceId/ProjectId to an device/instance from the application level
    1. Alternatively, we permit Instance snapshot view to see All Application snapshots (like a device can)
    2. Care point: When device snapshots "Show only Snapshots created by this device" is unchecked, ensure that it can access any orphaned snapshots (with matching ApplicationId)