[x] Multi app setup broker <--> controller #257 #253
[x] Unit tests #257
[x] Integration tests #257
[ ] General testing integration across ecosystem
[x] COS integration
broker <--> controller roles
Data flow between "controller" and "broker":
If an application is a controller, write to own databag/peer_cluster the controller-quorum-uris as soon as possible. These uris can be set at any given time, since they only depend on IP and unit id.
If an application is a broker, write to own databag/peer_cluster the cluster-uuid as soon as possible. At some point this will be a task for the cluster_manager instead of the broker.
Storage formatting
Formatting the storage before starting the cluster is a needed step on KRaft mode:
charmed-kafka.storage format --cluster-id $uuid -c server.properties --add-scram 'SCRAM-SHA-512=[name={user},password={password}]'
Running this command requires two things from the charm:
A well formed server.properties file. The command will look at log.dirs and metadata.log.dir and create the metadata files there. But the rest of the properties need to be, at least, format compliant. This implies that a broker also needs the controller-quorum-uris to be set on the file before calling format storages. So the broker needs to wait for the uris before this step.
Internal users. The admin and sync users are added to the metadata at the same time as the format is set. Since the internal users are created on the broker, the controller side needs to wait for the secret to be passed along.
Once those conditions are met, the service is able to start.
On unit-id / node.id handling
KRaft mode requires all nodes on the cluster (controller, broker) to have unique id numbers. For the time being, the node.id setup will be as follows:
jaas.file has been removed when using KRaft. Users are declared on the properties file, passed at init when formatting storages, or created through the kafka commands.
Adding the internal users (admin,sync) through the ACLS has been removed when running KRaft mode. These users are already on the cluster during formatting.
KRaft mode
TODO
broker,controller#235broker<-->controller#257 #253broker<-->controllerrolesData flow between "controller" and "broker":
controller, write to own databag/peer_cluster thecontroller-quorum-urisas soon as possible. These uris can be set at any given time, since they only depend on IP and unit id.broker, write to own databag/peer_cluster thecluster-uuidas soon as possible. At some point this will be a task for thecluster_managerinstead of the broker.Storage formatting
Formatting the storage before starting the cluster is a needed step on KRaft mode:
Running this command requires two things from the charm:
server.propertiesfile. The command will look atlog.dirsandmetadata.log.dirand create the metadata files there. But the rest of the properties need to be, at least, format compliant. This implies that a broker also needs thecontroller-quorum-uristo be set on the file before callingformat storages. So thebrokerneeds to wait for the uris before this step.adminandsyncusers are added to the metadata at the same time as the format is set. Since the internal users are created on thebroker, thecontrollerside needs to wait for the secret to be passed along.Once those conditions are met, the service is able to start.
On unit-id / node.id handling
KRaft mode requires all nodes on the cluster (controller, broker) to have unique id numbers. For the time being, the
node.idsetup will be as follows:broker:node.id = unit-id + 100controller:node.id = unit-idcontroller,broker:node.id = unit-id + 100Other relevant changes:
jaas.filehas been removed when using KRaft. Users are declared on the properties file, passed at init when formatting storages, or created through the kafka commands.admin,sync) through the ACLS has been removed when running KRaft mode. These users are already on the cluster during formatting.