[ ] Affected Issues have been mentioned in the Closing issues section
[ ] Documentation has been written/updated
[ ] PR title is ready for inclusion in changelog
Database Migrations
[x] If your PR contains a database migation, it MUST be the latest in date order alphabetically
This changes the way that project ids are stored against groups. It currently retains the function of adding the attributes in keycloak still for backwards compatibility with tools that use these attributes, but with a plan to remove this functionality once the other tools can be updated to use the project ids from the api-db.
The big changes are
store the group id to project id association in a table
store the group id to organization id association in a table
cache the group name to group id reference (purged when anything related to groups is performed)
cache the group response payload (purged when anything related to groups is performed)
cache the group members (purged when membership is modified)
a migration script that will retrieve existing groups/projects from keycloak and add them to the database
uses a custom mapper in keycloak to allow the opensearch token mapper to retrieve project ids from the api-db
Considerations
The biggest issue Lagoon has with using the attributes to store the project id assocation in keycloak is how we work out which groups a project has associated, as this requires us to retrieve all the groups from keycloak and iterate over them to check the attributes. This all groups query from keycloak is slow*number of groups, as more groups are added, slowness increases.
Adding these associations in the api-db, we can now query the api-db and get the information we need so we only need to go to keycloak if we absolutely have to.
While one or two calls to the allgroups query is usually fine, the way that a large number of lagoon queries work results in this query needing to be performed potentially multiple times, or a large number of times when users interact with the API in busy periods. This results in keycloak CPU utilization spiking. https://github.com/uselagoon/lagoon/pull/3397 was a first attempt at reducing how often the allgroups query was called, and it purged the data when groups were modified, but as the number of groups increases, this query gets slower and slower resulting in slow API responses or timeouts during busy periods
The lagoon-projects attribute on groups would need remain, at least for a period of time, as there are other systems that use these that would need to be refactored. While these other systems/tools use this attribute though, when a group is added/removed to a project, there is the potential due to the non-atomic nature of a lot of requests in Lagoon, that keycloak may not receive the attribute update, but the api-db does, or vice versa. This would result in incorrect access in either api or the other tools depending on the failure point. This not-atomic nature exists now, so this out of sync condition can and already does occur at times.
opensearch token mapper uses these attributes - This PR contains a custom java token mapper that will connect to the api-db to retrieve project IDs. This eliminates the need here for storing the project ids in keycloak.
More data is cached from keycloak to speed up performance of group and member based queries. As organizations exposes a part of Lagoon that has typically been hidden away from users for so long, and thus relatively unused. Organizations is very heavy on group and group member queries, it made sense to cache a lot of this where possible.
an entire group response is cached, this query when performed against keycloak is not that bad, but if a large number of groups are requested against keycloak directly, the response time can be long.
a groups membership is also cached, this sort of query can also be quite heavy as it can request a considerable amount of information from keycloak if multiple groups and their members are requested, pulling this from the cache is ideal.
any changes to a groups project ids, or a member/role change within a group results in that groups entire cache data being purged, and will be re-populated when it is requested next.
General Checklist
Database Migrations
This changes the way that project ids are stored against groups. It currently retains the function of adding the attributes in keycloak still for backwards compatibility with tools that use these attributes, but with a plan to remove this functionality once the other tools can be updated to use the project ids from the api-db.
The big changes are
Considerations
lagoon-projectsattribute on groups would need remain, at least for a period of time, as there are other systems that use these that would need to be refactored. While these other systems/tools use this attribute though, when a group is added/removed to a project, there is the potential due to the non-atomic nature of a lot of requests in Lagoon, that keycloak may not receive the attribute update, but the api-db does, or vice versa. This would result in incorrect access in either api or the other tools depending on the failure point. This not-atomic nature exists now, so this out of sync condition can and already does occur at times.