Proposal to separate the live payments dashboard from the Toolbox repository and deployment (draft, optional Firebreak candidate)

[sfount]

The live payments dashboard was added to Toolbox during a Firebreak in 2019. The dashboard has been used internally to highlight that alongside any current day challenges and work in progress, the platform runs smoothly and is relied on by services across government.

At the time bundling the dashboard into internal tooling was justified (as explained further down) and allowed it to be deployed quickly.

Separating the dashboard code and hosting would address a number of pain points:

• Allow the dashboard to be run when not connected to the office WiFi or the VPN. Connection issues with the VPN, both in the GDS offices and at external events, have caused unreliable behaviour that leads to it not working
• Remove any direct calls to payments infrastructure. As the code was running internally anyway, the polling mechanism relies on calling Pay's microservices which limits hosting and deployment options

Additional operational implications:

• Removing the browser build step would significantly increase the Toolbox build speed and improve developer experience (minutes -> seconds)
• Removing many dependencies that are not used anywhere else in Pay. These dependencies will likely be lesser known by Pay engineers and require a lot more effort to upgrade or patch
• Remove code from Toolbox that is only there to support the dashboard and may be harder to support
• Allow support and maintenance for the dashboard to be removed without impacting the critical path of Toolbox supporting the platform
• Make the hosting and authorisation for non-critical tools more consistent with other Pay repositories (for example the team manual)
• A number of other teams within GDS have shown interest in seeing how the dashboard works and thinking about it in the context of their service. Separating the code and documentation would make it easier to share and give an opportunity to clean up the code

Context

The dashboard was originally designed for use in the office, running on small TV screens in the GOV.UK Pay team area. This meant that:

• The machines loading the dashboard were always connected to the office WiFi network and could access internal tooling
• The GOV.UK Pay team who knew what it represented and could talk about Pay to those interested, they could also fix anything that went wrong
• The dashboard would only ever be loaded once in the team area, how well it scaled fetching the information needed to run it was not a priority
• In late 2019 the GOV.UK Pay platform processed orders of magnitude less payment traffic, the animations driven by real time events were slow enough to be legible and not too visually distracting

Proposed method

• Separate the frontend code (React) required to run the dashboard from the Toolbox repository into its own repository alphagov/pay-live-payments-dashboard
• Host the page produced by alphagov/pay-live-payments-dashboard using GitHub pages, similar to the Pay team manual
  • For parity with existing access control, limit the pages access to GitHub or Google SSO. Access control can then be flexibly changed as required
• Use AWS S3 to store and act as the backend for the limited information needed to display the dashboard
  • A scheduled lambda can fetch information from Pay microservices and store it in an AWS S3 bucket in a CSV format
    • The data in the AWS S3 bucket could be configured to expire after X hours/ days, this should allow it to remain timely and never grow in size and cost
    • While this schedule could be run frequently to continue showing "live" data, a potential simplification would be to replay what happened yesterday on the dashboard. That could mean only running the schedule once at the end of a day so that the dashboard can use the data for it tomorrow
  • The React frontend can then fetch this information from the AWS S3 bucket rather than calling Pay services

    Optional extension

    When the dashboard has been hosted on larger screens in the office, it has been reported that the speed of the animations (driven by the speed of payments) can be distracting . Time and people dependant we could also address this while taking the time to refactor and organise the code in a separate repository.

Previous proposals have been to:

• change the payment event cards to allow them to smoothly "scroll" between positions rather than flashing and being replaced
• only show payment success event messages in the feed, currently it displays when a payment is created, when card details are entered and when a payment is completed. Each payment therefore is represented 3 times in the feed increasing animation speed
• use sampling over the last X seconds (between 1 and 5?) to determine which payment to show rather than showing everything, this would allow for a consistent ticket that doesn't change in speed and can be slowed down or sped up (to "live")

Current architecture

Dashboard fetches data in real time from Pay services

sequenceDiagram
    Dashboard frontend (Toolbox) -> Toolbox server: requests information on services
    Toolbox server -> Adminusers microservice: requests information on services
    Dashboard frontend (Toolbox) -> Toolbox server: requests aggregate stats for the day so far
    Toolbox server -> Ledger microservice: requests aggregate stats for the day so far
    loop Every 5 seconds
        Dashboard frontend (Toolbox) -> Toolbox server: requests events data for the last few seconds
        Toolbox server -> Ledger microservice: requests events data for the last few seconds
    end

Proposed architecture

Dashboard fetches data from S3

sequenceDiagram
    Dashboard frontend (GitHub Pages) -> S3: Fetch information on services and aggregate stats
    Loop Every 10 minutes
        Dashboard frontend (GitHub Pages) -> S3: Fetch event information on payments
    end

Independently a scheduled task fetches data from Pay services and stores it in the appropriate format for S3

sequenceDiagram
    loop Once at the end of the day
        Scheduled lambda -> Ledger microservice: Get up to date information about payment events
        Scheduled lambda -> Adminusers microservce: Get up to date information about services
        Scheduled lambda ->> S3: Write information to bucket
    end

[katstevens]

I'm assuming this proposal means Pay still owns the artefacts here (repo, lambda, S3 bucket, IAM policies to allow everything to happen)?

[katstevens]

We'd also have make sure we only store essential payment metadata in the S3 bucket (amount, service, status, payment type - no personal data or reference numbers). Don't know if the current API endpoint Toolbox uses returns anything else. We need to remove any risk of incidental pollution that could end up in the bucket.

[sfount]

Yeah Pay would own those.

I suppose one way of looking at it is shifting the complexity from internal real-time requests over to ETL infrastructure to put just the right amount data in the right place at the right time. (So less code to maintain in Toolbox, and less things hitting internal microservices, but more infrastructure)

Agree what ends up in the bucket would be a big part of if it could work or not. At the moment the only thing Toolbox gets is service (UUID), status, amount, time but as it has always been IP locked doing some threat modelling on whats stored there would be needed.

Maybe one way of approaching that would be to think through what would be needed to no longer rely on internal API calls but continue with the IP restrictions until the threat modelling had been done.

[Beth-Brown]

I'd really like to see the accessibility issue with the speed on updates / flashing green completed payments resolved as part of this work.

I love this proposal - its a great way for us to be able to use the payments dashboard in external events or in product demos more easily in future.

[Beth-Brown]

I'd like to see this work broken up into sizeable chunks that we can prioritise when required. This project would be much bigger than one firebreak week right now, so breaking it down into some valuable deliverables will really help us to achieve incremental progress over time and manage our expectations over how long this work is likely to take