search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

automatic memory configuration under k8s #192

Closed javacruft closed 1 year ago

javacruft commented 1 year ago

Steps to reproduce

juju deploy --constraints mem=1G --channel edge mysql-k8s mysql-small

Expected behavior

The deployed units should tune their memory configuration to the provided constraint - in this case 1GB.

If no constraint is provided the charm needs todo something consistent and capped in some way across all units in the app.

Actual behavior

The deployed units tune their memory configuration based on the total memory of the machine hosting K8S; this results in to much memory being configured given the K8S provided constraint - at 1G this stops the mysql init process, at 2G the mysql init process completes but at some point in time the ceiling will be hit by the mysqld process.

If no constraint is provided the innodb buffer pool sizes are tuned without any limits - on a large machine/under k8s this does not make sense as there may be many instances of mysql on the same K8S deployment and they all tune based on the assumption they can take all of the RAM.

I'd proposed that if a constraint is provided, then that is used for tuning memory configuration; if no constraint is provided then an automatic calculation is made but with some sort of sensible upper bound - this is the pattern we followed for the original Percona Cluster and MySQL InnoDB Cluster charms - reference charm-helpers:

https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/database/mysql.py#L689

Versions

Operating system: 22.04 Juju CLI: 3.2/candidate Juju agent: 3.2-beta1 Charm revision: 51 microk8s: 1.26/stable

Log output

juju debug-log --replay ```shell unit-mysql-small-0: 09:24:30 INFO juju.cmd running containerAgent [3.2-beta1 ca8984c1fb29fc9eaf388e414b54acc60937e7d8 gc go1.20.2] unit-mysql-small-0: 09:24:30 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-small-0: 09:24:30 INFO juju.worker.upgradesteps upgrade steps for 3.2-beta1 have already been run. unit-mysql-small-0: 09:24:30 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301 unit-mysql-small-0: 09:24:30 INFO juju.api cannot resolve "controller-service.controller-sunbeam-microk8s-localhost.svc.cluster.local": lookup controller-service.controller-sunbeam-microk8s-localhost.svc.cluster.local: operation was canceled unit-mysql-small-0: 09:24:30 INFO juju.api connection established to "wss://10.152.183.89:17070/model/15048031-65e3-4485-8fd5-d078a60ef9d1/api" unit-mysql-small-0: 09:24:30 INFO juju.worker.apicaller [150480] "unit-mysql-small-0" successfully connected to "10.152.183.89:17070" unit-mysql-small-0: 09:24:30 INFO juju.api connection established to "wss://controller-service.controller-sunbeam-microk8s-localhost.svc.cluster.local:17070/model/15048031-65e3-4485-8fd5-d078a60ef9d1/api" unit-mysql-small-0: 09:24:30 INFO juju.worker.apicaller [150480] "unit-mysql-small-0" successfully connected to "controller-service.controller-sunbeam-microk8s-localhost.svc.cluster.local:17070" unit-mysql-small-0: 09:24:30 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-small-0: 09:24:30 INFO juju.worker.logger logger worker started unit-mysql-small-0: 09:24:30 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-small-0: 09:24:30 INFO juju.worker.leadership mysql-small/0 promoted to leadership of mysql-small unit-mysql-small-0: 09:24:30 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-small-0: 09:24:30 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-small-0: 09:24:30 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-small-0 unit-mysql-small-0: 09:24:30 INFO juju.worker.uniter unit "mysql-small/0" started unit-mysql-small-0: 09:24:30 INFO juju.worker.uniter resuming charm install unit-mysql-small-0: 09:24:30 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-k8s-51 from API server unit-mysql-small-0: 09:24:30 INFO juju.downloader downloading from ch:amd64/jammy/mysql-k8s-51 unit-mysql-small-0: 09:24:30 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-k8s-51") unit-mysql-small-0: 09:24:30 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-k8s-51") unit-mysql-small-0: 09:24:40 INFO juju.worker.uniter hooks are retried true unit-mysql-small-0: 09:24:40 INFO juju.worker.uniter found queued "install" hook unit-mysql-small-0: 09:24:41 INFO unit.mysql-small/0.juju-log Running legacy hooks/install. unit-mysql-small-0: 09:24:42 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:42 INFO juju.worker.uniter.operation ran "database-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:42 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-small-0: 09:24:43 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:44 INFO juju.worker.uniter.operation ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:44 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:45 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:45 INFO juju.worker.uniter found queued "start" hook unit-mysql-small-0: 09:24:46 INFO unit.mysql-small/0.juju-log Running legacy hooks/start. unit-mysql-small-0: 09:24:46 INFO unit.mysql-small/0.juju-log Retrieving the total memory of the server unit-mysql-small-0: 09:24:48 ERROR unit.mysql-small/0.juju-log Exited with code 137. Stderr: unit-mysql-small-0: 09:24:49 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:24:49 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-small-0: 09:30:26 WARNING unit.mysql-small/0.juju-log No relation: certificates unit-mysql-small-0: 09:30:26 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) ```

Additional context

juju ssh --container mysql mysql-small/0 "cat /etc/mysql/conf.d/z-custom.cnf" ```shell [mysqld] report_host = mysql-small-0.mysql-small-endpoints.mysql.svc.cluster.local innodb_buffer_pool_size = 12616466432 # 11.75GB innodb_buffer_pool_chunk_size = 1577058304 ```
taurus-forever commented 1 year ago

@javacruft please switch your tests from latest/edge to 8.0/edge track. The fix should be there.

P.S. @paulomach thank you for the superfast fix! P.P.S The fix should also be available in 8.0/candidate this week.