🗃📦🌳 Feature/usage and quotas

What's new

This PR introduces Usage and Quotas. A Quota defines limits of consumption of key aspects of a conference (including all subconferences).

Once the limits have been proven in production we will then layer our checkout flow on top, including UI for displaying quotas and usage, and ways to purchase additional quota.

Quotas

Quotas (/limits) are:

whether streaming events are allowed
whether video-chat events are allowed
time limits for the total consumption of streaming events, video-chat events and video-chat outside of events
time limits for the individual durations of streaming and video-chat events
time limits for support meetings (manually tracked)
maximum number of subconferences
maximum number of registrants
maximum number of streaming program rooms, non-streaming program rooms and public social rooms
maximum number of content items
maximum number of media and non-media elements per sponsor content item
maximum number of media and non-media elements per non-sponsor content item

Measuring usage

Consumption of streaming and video-chat event minutes is measured by regular calls to the idempotent updateEventUsage Postgres function (via the GQL API). Consumption of video-chat outside of events is measured by tracking registrant's connections to video chats (via Redis) and regular aggregating the totals via the Actions service.

Enforcing the quota

Postgres triggers are used to enforce the remaining quota on the relevant tables. The Actions service checks video-chat consumption prior to granting a connection token.

Other major changes

The merge of the changes to presence tracking to use ping-based rather than perfect-connection-tracking.
- This requires Redis to be flushed of all PresenceList:* keys because the type has changed from hashset to ordered set
The removal of the room.Participant table
- Room participants are now tracked in Redis.
- Vonage connections are also now tracked in Redis (though Vonage Streams continue to be tracked in Postgres)
- This enables efficient tracking of connections, connection duration and total consumption
- This enables more efficient fetching of the list of people connected to a room
- This will be significantly more reliable even if the main Postgres database suffers load issues at the start of an event (when everyone is joining).
- This will reduce load on Postgres (inserts and deletes coupled with many selects was creating a perfect storm for Postgres)
Enabling of server-side query caching in Hasura (by adding @cached to many of the frontend queries)
- Our work on the next branch (recently merged) was heavily focused on making user's queries overlap, even in the Hasura session variable values.
- This work is what enables us to make use of Hasura's server-side caching
  1. As a result of ready, low-impact access to the count of connected participants to a room (via Redis - see (2)), the Capacity field of Rooms is now respected when someone tries to connect to a room.
    - The corresponding field in the Admin UI is now shown again
  2. Program rooms are now either streaming rooms or non-streaming rooms.
    - This is determined by whether any streaming event is scheduled in a room
    - The default video chat is not shown in a streaming room
    - Postgres insert/update triggers prevent the formation of events from different mode-classes (streaming vs non-streaming) in the same room

Other minor changes

Commenting out the Shuffle Queue Algorithm selection in the Admin UI, since this has never yet been finished
Renamed "Shuffle" to "Networking" in the admin landing page

Deployment

Instructions for how to deploy these changes to production. Please tick (i.e. put an 'x' in) the boxes for the actions that are necessary.

[x] Apply migrations to Hasura
[x] Apply metadata to Hasura
[x] Re-deploy frontend
[x] Re-deploy all services
[x] Flush real-time service state (Redis)
- Clears PresenceList:* keys which were Sets and are now Sorted Sets. Without clearing, you will see "Wrong Type" errors in the realtime service.
[x] Add Cron2Go job for maintaining presence lists

Testing

This is the first time we are introducing proper automated testing. The tests are contained within /tests/usage-and-quotas and can be executed using pnpm test (and various other options - see package file).

Tests are built using the Vitest framework and require an env file and Hasura to be running locally. Each test executes sequentially due to the problem of certain conflicting database update calls that apply to all conferences at once (and so would affect concurrent tests at random).

Tests cover all aspects of the changes within the database: the tables, views, restrictive triggers and update usage function.

Four of the tests are skipped by default as they run for 60-seconds each, which would slow down routine testing. They should be enabled for testing prior to submitting a PR.

clowdr-app / clowdr