FlowFuse / flowfuse

Connect, collect, transform, visualise, and interact with your Industrial Data in a single platform. Use FlowFuse to manage, scale and secure your Node-RED solutions.

Other

278 stars 63 forks source link

Description

Part of #2782

The Forge App runs some tasks on a regular schedule as part of its own house keeping. For example

Reporting usage telemetry
Managing Trial life-cycle
Other database housekeeping

Depending on the specific task, we may not want it running in multiple places at once. For example, the Trial Task isn't transactional - it gets the list of all subscriptions to operate on first, then does its thing. It would be possible for two instances to run in parallel, both get the same list and then both try to do the work.

Possible Solutions

Specific solutions will depend on the needs of the individual tasks and they will have to be considered on a case-by-case basis.

Some approaches will be:

Rewrite tasks to be transactional and concurrency-safe
Add jitter to the scheduling of the tasks so they don't run at the same time
Add some sort of quorum holder capability so an instance takes a primary role to run the tasks... lots of dragons to be had here.

Epic/Story

2782

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

We currently have 4 timerTasks to consider.

expireTokens - runs @daily, removes any tokens that have expired. No real issue this running repeatedly - just needs some jitter as @daily translates to 0 0 * * * and we wouldn't want all instances running at the same time.
licenseCheck - runs '0 2 * * *'. Checks license expiry and emails admins if nearing expiry date and/or expired. Don't want to send multiple emails each day.
licenseOverage - runs @weekly (0 0 * * 0). Generates audit log messages if the platform is over license terms. Don't want duplicate audit log entries.
telemetryMetrics - runs 10s after startup and at a randomly picked minute/hour each day. Not ideal to have it run multiple times, but at least with the randomly picked reporting time it isn't too bad,

To simplify this task I think we only need to address the scheduling of the expireTokens task.

[ ] https://github.com/FlowFuse/flowfuse/pull/3327

As for the others:

licenseCheck/licenseOverage - this is an edge case I don't think we need to prioritise fixing today. It would require an platform running with multiple instances and an expiring license. Duplicate emails/audit log entries are not ideal, but not show stopping.
telemetryMetrics - we can live with multiple reports each day