BEP: Backstage Notifications System

[Rugvip]

🔖 Feature description

The Backstage Notifications System provides a way for Backstage plugins to send notifications to users. The notifications are displayed in the Backstage frontend UI, but future extensions can allow them to be sent via external channels such as email. The system includes a push mechanism to make sure that users receive notifications immediately, as well as persistence of notification history and read status.

🎤 Context

beps/0001-notifications-system

✌️ Possible Implementation

No response

👀 Have you spent some time to check if this feature request has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

Are you willing to submit PR?

None

[mareklibra]

To support asynchronous tasks, external systems need to be able to create new notifications.

In the early implementation of Notifications, this is possible by reaching the REST endpoint exposed by the Notifications backend, optionally authenticated.

In the BEP , this can be achieved by (custom) building yet another BE plugin exposing another REST endpoint of the same or very similar API which enforces authentication and passing-through data.

Such a composition increases complexity. What about allowing external systems (non BE plugins) to reach the notifications REST endpoint and enable/disable such flow by configuration?

[Rugvip]

@mareklibra the BEP suggests that the API for sending notifications is protected by service-to-service auth. That will restrict access from users, but other backend services are still able to call the API, regardless of whether they're backstage backend plugins or external systems. With our current implementation of service-to-service auth all that is required is to generate a valid JWT using the configured backend secret, although this is something we plan to evolve and improve upon separately as part of #15999

[mareklibra]

A notification should further contain:

• title Human readable text shortly describing the notifications

• priority One of predefined values to help with presentation of the messages or filtering Values: Critical, High, Normal, Low

• icon - optional To improve UX. A string referencing an icon name from MUI icon library of a defined version. If missing, it will be determined from the priority

• topic A string value helping to group notifications of particular context Use cases:

  • several notifications emitted by an asynchronous external task can be grouped by a single topic
  • a BE plugin can group several related messages to a particular processing, i.e. asynchronous progress monitoring

• time-stamp of creation

  • to filter messages or prune the DB eventually or detect "long unread notifications"

• origin Identifications of the creator, i.e. the BE plugin

The payload should cover:

• additional text Optional in-depth content of the notification to be displayed to the user

• a list of URLs and their descriptions Use cases:

  • an external system can provide more details related to the notification
  • an action in an external system is requested within asynchronous task
  • a BE plugin can provide link to other part of the Backstage UI (i.e. to the Catalog)

[Rugvip]

@mareklibra could you open a PR to add that to the design details section in the BEP once the initial PR is merged?

Main initial feedback on the list is that I think it's important to separate between input and output schema. In the details section I mention the possibility of adding NotificationParameters and Notification as types - which I think would be a good way forward here. For example the creation time stamp would not be part of the input parameters, only the output notification as read from the API.

[drodil]

I think most of the things in @mareklibra comment can be added later. If we just get the basic functionality working, it's then possible to add more stuff for UX etc. I am doing some refactoring based on the BEP to #21889.

[Rugvip]

@drodil I'd like is to try to figure out the overall strategy for managing the notifications schema. I agree finer details like whether there's an icon link or not could wait, but it would be good if we could get to the point where it's obvious where it would be added and how it is consumed. Discussing a couple of concrete examples is a good way to get there, so I think it makes sense to dive into some depth still.

[freben]

I think it may be important to be able to segregate generation-time (the producing system decides this, this is probably an optional parameter and is present both in the input and output schema) and reception-time (the notifications backend decides this, and it's always present with a value only in the output schema).

This becomes extra important if the producing system buffers events or generate the events based on an external truth (such as a timestamp from a third party system).

[freben]

I have a question about the use of the signal backend's use of a signals topic. It's mentioned that the signal backend might get a broader set of use cases in the future. But let's say that the catalog backend wants to start publishing "entity with ref k:ns/n has finished stitching with changes". It wants to broadcast that fact to the world, definitely not only for the purpose of notifications. How will it do this? Will it send two events - one general on a topic that it controls, and a narrower one also on the signals topic? Or is there some form of a bridge between the two topics then? How does this look on the consuming end on the frontend - will the entity page be able to essentially send a SUBSCRIBE on the entity ref and get a stream of events relevant to it?

I know this is much to ask at this point, but it may be relevant to at least have some form of a plan for this, and whether some more labels-based scheme could have served those use cases batter.

[Rugvip]

@freben tbh I think the catalog should just publish a separate signals event that matches what has been implemented in the catalog frontend. It keeps things a lot simpler, and the main thing we need to make sure is that sending signals without anyone listening is cheap, which is something we need to solve for anyway.

[Rugvip]

Lots of good discussion in https://github.com/backstage/backstage/pull/22641 👍

List of unresolved threads:

• [ ] How to generate in-app links for notifications: https://github.com/backstage/backstage/pull/22641#discussion_r1475794686
• [ ] Whether notifications should be immutable: https://github.com/backstage/backstage/pull/22641#discussion_r1475857522
• [ ] Removing image URL: https://github.com/backstage/backstage/pull/22641#discussion_r1475858757
  • I think we agree on this one, but the proposal still references image URLs in text
• [ ] Handling of multiple links: https://github.com/backstage/backstage/pull/22641#discussion_r1475860873
  • As far as I can tell we all agree that some way to represent multiple links can be useful, but what do we need to have built-in as part of the core schema?
• [ ] Notification as Tasks: https://github.com/backstage/backstage/pull/22641#discussion_r1475927485

Let's avoid diving into discussions on any of these topics in this issue, please instead open a PR towards the BEP and link the PR here, then we'll have the detailed discussions there.

[drodil]

Further things to consider adding to the system

• [ ] User specific notification settings (ways to get notifications from core and processors, topics, etc.)
• [ ] Adding support for analytics using analyticsApi
• [ ] Automatic cleaning of old notifications based on configuration

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[drodil]

Not stale, still lots of TODOs

[Rugvip]

I want to highlight this discussion around structured metadata for notifications, where we also discussed moving some of the existing fields to be metadata instead, such as links, icons, or anything that you don't need to be able to query notifications by. Any general thoughts?