[Epic] System flows functionality

[anna-geller]

Changes in the UI

Namespaces Overview

The system namespace should be pinned at the top.

https://www.figma.com/design/ew0uXk0NRXJ2NBBJTNe2n1/01_UI?node-id=681-5355&t=CM3hskgx8LI6ehf7-4

Tabs of the System Namespace

When you then navigate into that Namespace, the first tab displayed there should be "System Blueprints" to make them easily accessible. This will display the same Blueprints table but filtered for the System tag.

https://www.figma.com/design/ew0uXk0NRXJ2NBBJTNe2n1/01_UI?node-id=2428-31298&t=CM3hskgx8LI6ehf7-4

The main text that can be displayed at the top:

Keep your platform in check with system flows. Automate maintenance tasks, from failure alerts to automated cleanups.

New filter on the Flows and Executions pages + on the main Dashboard

Both the Flows and Executions pages need an additional multi-select field allowing users to select User Flows/Executions, System Flows/Executions, or both.

By default, only the User Flows/Executions should be preselected.

Executions page

• User Executions
• System Executions

Flows page

• User Flows
• System Flows

Main Dashboard

• User
• System

---

Remove the warning

The warning:

"The system namespace is reserved for background workflows intended to perform routine tasks such as sending alerts and purging logs. Please use another namespace name."

can now be deleted.

---

RBAC

System namespaces are open by default. To restrict them, you can restrict their use via system namespace permissions allocated only to Admins and assigning company.* namespaces to everyone else in the company.

---

Add blueprints

This is less for developers and more for product (@anna-geller and @wrussell1999) — we need to prepare many System Blueprints to be displayed in the System Blueprints tab. Those will be maintenance flows that help accomplish the following use cases:

• [x] Purge old logs = log retention https://github.com/kestra-io/kestra-ee/issues/1347
• [x] Purge old executions and internal storage files (inputs, outputs, data passed between tasks, trigger data, trigger logs) -- https://kestra.io/blueprints/trigger/234-purge-execution-data-including-logs-metrics-and-outputs-on-a-schedule (related to https://github.com/kestra-io/kestra/issues/4206)
• [x] Alert on failure via Flow trigger reacting to Executions in a Failed state (and optionally also Warning state)
• [x] Create automated support tickets (via GitHub issue, JIRA, Zendsk, Linear etc.) on the failure of some critical flows
• [x] CI/CD: Sync code from Git or push code to Git — using kestra's Git plugin

[anna-geller]

@wrussell1999, I started adding system flows https://kestra.io/blueprints/system. Let's continue together, checklist of what we need:

• [x] purge logs and internal storage data
• [x] alert on failure using Slack
• [x] alert on failure using Teams
• [x] alert on failure using Discord
• [x] alert on failure using Sentry
• [x] alert on failure using Zenduty
• [x] alert on failure using Email
• [x] alert on failure using Gmail (I think Gmail is so common that it's worth adding a dedicated blueprint with a Setup section explaining how to set up Gmail, maybe even with embedded screenshots in Strapi?)

[MilosPaunovic]

This issue has now been completed, or at least the part above Add blueprints section in the main description.

Pull Requests are merged for both OSS and EE, but it still isn't closed as it would be a good idea for @Ben8t or someone else give it another test with fresh set of eyes.

[Ben8t]

Waiting for Docker images to build, I'll take a look thanks! 👍

[Ben8t]

@anna-geller one wondering in the meantime:

System namespaces are open by default. To restrict them, you can restrict their use via system namespace permissions allocated only to Admins and assigning company.* namespaces to everyone else in the company.

Do secrets stored in system namespaces will be shared with any other "root" namespace? Put it differently, do users need to duplicate secrets with system and <root_namespace>.<some_project>.<...> ?

Example: I've a system flow to send alter via Slack (some SLACK TOKEN secret set up in system namespace) and in my root namespaces (where I have my actual projects) I also need to have that secrets for other Slack related tasks 🤔

[anna-geller]

they will need to be in system namespace

[MilosPaunovic]

CI issues are amended, our kafka preview is deployed, so this is now safe for testing @Ben8t.

[anna-geller]

LGTM, well done! I opened a separate remaining issue https://github.com/kestra-io/kestra/issues/4873

[Ben8t]

@MilosPaunovic are you sure everything deployed ?

Can't see any system namespace in kafka-preview or pulled develop image 🤔

Will wait a bit probably ?

[MilosPaunovic]

That sound like a browser-cache problem, try hard refreshing/changing browsers, as on kafka I see it properly

[Ben8t]

Found the trick: if no flow in system namespaces, the namespace doesn't show up (logic!)

current discussion for ref