High availability (HA) / K8s StatefulSet compatibility

[brettaufheber]

To ensure compatibility for Kubernetes StatefulSet mechanism, it should be possible for a master node to perform a sync to a Crowd server while the slave nodes are limited to a read-only access to the shared database.

[brettaufheber]

Overview for joining a cluster: https://doc.akka.io/docs/akka/current/typed/cluster.html#cluster-membership-api

Boot cluster without the configuration of seed nodes: https://doc.akka.io/docs/akka/current/additional/operations.html#cluster-bootstrap

Cluster singleton pattern: https://doc.akka.io/docs/akka/current/typed/cluster-singleton.html

[peterhoepfl]

To solve the reqiurement - when running multiple instances of the application only one instance shall perform the sync operations between Crowd and the database - I suggest a much simpler solution: To ensure that only one of the scheduled tasks (one per instance) does the actual work we should use a logical or session database lock, in Postgres they are called advisory locks. When the task starts he tries to get the advisory lock - the lock mechanism ensures that only one task will get the lock. If he succeeds he keeps the lock until the sync is finished. The other tasks will fail in getting the lock and won't perform the sync. This design pattern is f.e. also used in the liquibase extension "sessionlock" and I use that pattern since many many years to prevent scheduled tasks/jobs running parallel when multiple instances of a job exist.

[peterhoepfl]

The scheduled task has fixed initial delay and intervall, this must be made configurable. It should be possible to disable the databaselocks by config property.

[peterhoepfl]

With version 7.2.0 the dblock feature is released. Solution as described above is implemented.