search

canonical / mysql-router-k8s-operator

Mysql router operator charm for kubernetes
https://charmhub.io/mysql-router-k8s
Apache License 2.0
2 stars 7 forks source link

Blocked after deploy: Failed to create k8s service #22

Closed carlcsaposs-canonical closed 1 year ago

carlcsaposs-canonical commented 1 year ago

To reproduce: juju deploy mysql-router-k8s --channel edge --trust Charmhub revision: 7 microk8s version: MicroK8s v1.26.0 revision 4390 juju agent version: 2.9.38, reproduced on 2.9.29 Ubuntu version: 22.04

$ juju deploy mysql-router-k8s --channel edge --trust
Located charm "mysql-router-k8s" in charm-hub, revision 7
Deploying "mysql-router-k8s" from charm-hub charm "mysql-router-k8s", revision 7 in channel edge on jammy

Error in debug log:

unit-mysql-router-k8s-0: 15:49:55 ERROR unit.mysql-router-k8s/0.juju-log Failed to create k8s service
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/api/v1/namespaces/controller/pods/mysql-router-k8s-0'
For more information check: https://httpstatuses.com/404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 225, in _on_leader_elected
    self._create_service(f"{self.app.name}-read-only", 6447)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 94, in _create_service
    pod0 = client.get(
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/client.py", line 108, in get
    return self._client.request("get", res=res, name=name, namespace=namespace)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: pods "mysql-router-k8s-0" not found
Full debug-log ``` $ juju debug-log --replay controller-0: 15:35:56 INFO juju.cmd running jujud [2.9.38 6d211be0d72d6f4d625c61c7c4ddb4e9325226c8 gc go1.19.4] controller-0: 15:35:56 INFO juju.worker.upgradesteps upgrade steps for 2.9.38 have already been run. controller-0: 15:35:56 INFO juju.state using client-side transactions controller-0: 15:35:56 INFO juju.state starting standard state workers controller-0: 15:35:56 INFO juju.state creating cloud image metadata storage controller-0: 15:35:56 INFO juju.state started state for model-9fa2c6a8-d23d-48b1-8124-bfc190891886 successfully controller-0: 15:35:56 INFO juju.cmd.jujud juju database opened controller-0: 15:35:56 INFO juju.state using client-side transactions controller-0: 15:35:56 INFO juju.worker.lease.raft Using lease client type "pubsub" for raft lease transport controller-0: 15:35:56 INFO juju.state.allwatcher allwatcher loaded for model "9fa2c6a8-d23d-48b1-8124-bfc190891886" in 6.50152ms controller-0: 15:35:56 INFO juju.state ensuring logs collection for 9fa2c6a8-d23d-48b1-8124-bfc190891886, capped at 20 MiB controller-0: 15:35:56 INFO juju.state starting standard state workers controller-0: 15:35:56 INFO juju.state creating cloud image metadata storage controller-0: 15:35:56 INFO juju.state started state for model-9fa2c6a8-d23d-48b1-8124-bfc190891886 successfully controller-0: 15:35:56 INFO juju.worker.httpserver listening on "[::]:17070" controller-0: 15:35:56 INFO juju.api connection established to "wss://localhost:17070/model/9fa2c6a8-d23d-48b1-8124-bfc190891886/api" controller-0: 15:35:56 INFO juju.worker.apicaller [9fa2c6] "controller-0" successfully connected to "localhost:17070" controller-0: 15:35:56 INFO juju.apiserver.common setting mongo password for "controller-0" controller-0: 15:35:56 INFO juju.apiserver.common setting password for "controller-0" controller-0: 15:35:56 INFO juju.api connection established to "wss://localhost:17070/model/9fa2c6a8-d23d-48b1-8124-bfc190891886/api" controller-0: 15:35:56 INFO juju.worker.apicaller [9fa2c6] "controller-0" successfully connected to "localhost:17070" controller-0: 15:35:56 INFO juju.apiserver.common setting mongo password for "controller-0" controller-0: 15:35:56 INFO juju.apiserver.common setting password for "controller-0" controller-0: 15:35:57 INFO juju.worker.caasupgrader abort check blocked until version event received controller-0: 15:35:57 INFO juju.worker.caasupgrader unblocking abort check controller-0: 15:35:57 INFO juju.api connection established to "wss://localhost:17070/model/9fa2c6a8-d23d-48b1-8124-bfc190891886/api" controller-0: 15:35:57 INFO juju.worker.apicaller [9fa2c6] "controller-0" successfully connected to "localhost:17070" controller-0: 15:35:57 INFO juju.worker.migrationminion migration phase is now: NONE controller-0: 15:35:57 INFO juju.agent removing system identity file controller-0: 15:35:57 INFO juju.worker.logger logger worker started controller-0: 15:35:57 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" controller-0: 15:35:58 INFO juju.worker.raft.raftclusterer server "0" (Voter) address changed from "localhost" to "10.152.183.186:17070" controller-0: 15:36:01 WARNING juju.core.raftlease response timeout waiting for Command(ver: 1, op: claim, ns: singular-controller, model: 9fa2c6, lease: 95036c2a-22d3-4e44-8d12-fe3d04ca7b55, holder: controller-0) to be processed controller-0: 15:36:01 INFO juju.worker.raft.raftforwarder claiming lease "9fa2c6a8-d23d-48b1-8124-bfc190891886:singular-controller#95036c2a-22d3-4e44-8d12-fe3d04ca7b55#" for "controller-0" controller-0: 15:36:02 WARNING juju.core.raftlease response timeout waiting for Command(ver: 1, op: claim, ns: singular-controller, model: 9fa2c6, lease: 9fa2c6a8-d23d-48b1-8124-bfc190891886, holder: controller-0) to be processed controller-0: 15:36:02 INFO juju.worker.raft.raftforwarder claiming lease "9fa2c6a8-d23d-48b1-8124-bfc190891886:singular-controller#9fa2c6a8-d23d-48b1-8124-bfc190891886#" for "controller-0" controller-0: 15:36:02 INFO juju.worker.logforwarder config change - log forwarding not enabled controller-0: 15:36:02 INFO juju.worker.logger logger worker started controller-0: 15:36:02 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for controller (9fa2c6a8-d23d-48b1-8124-bfc190891886) controller-0: 15:36:02 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for controller (9fa2c6a8-d23d-48b1-8124-bfc190891886) controller-0: 15:36:02 INFO juju.apiserver.common setting password for "model-9fa2c6a8-d23d-48b1-8124-bfc190891886" model-9fa2c6a8-d23d-48b1-8124-bfc190891886: 15:36:04 INFO juju.worker.caasupgrader abort check blocked until version event received model-9fa2c6a8-d23d-48b1-8124-bfc190891886: 15:36:04 INFO juju.worker.caasupgrader unblocking abort check model-9fa2c6a8-d23d-48b1-8124-bfc190891886: 15:36:04 INFO juju.worker.muxhttpserver starting http server on [::]:17071 model-9fa2c6a8-d23d-48b1-8124-bfc190891886: 15:36:04 INFO juju.worker.caasadmission ensuring model k8s webhook configurations controller-0: 15:49:37 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-router-k8s" controller-0: 15:49:37 INFO juju.apiserver.common setting password for "application-mysql-router-k8s" controller-0: 15:49:41 INFO juju.apiserver.connection agent login: application-mysql-router-k8s for 9fa2c6a8-d23d-48b1-8124-bfc190891886 controller-0: 15:49:41 INFO juju.apiserver.connection agent disconnected: application-mysql-router-k8s for 9fa2c6a8-d23d-48b1-8124-bfc190891886 controller-0: 15:49:46 INFO juju.apiserver.connection agent login: unit-mysql-router-k8s-0 for 9fa2c6a8-d23d-48b1-8124-bfc190891886 unit-mysql-router-k8s-0: 15:49:46 INFO juju.cmd running containerAgent [2.9.38 6d211be0d72d6f4d625c61c7c4ddb4e9325226c8 gc go1.19.4] unit-mysql-router-k8s-0: 15:49:46 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.upgradesteps upgrade steps for 2.9.38 have already been run. unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.probehttpserver starting http server on [::]:65301 unit-mysql-router-k8s-0: 15:49:46 INFO juju.api cannot resolve "controller-service.controller-micro2.svc.cluster.local": lookup controller-service.controller-micro2.svc.cluster.local: operation was canceled unit-mysql-router-k8s-0: 15:49:46 INFO juju.api connection established to "wss://10.152.183.186:17070/model/9fa2c6a8-d23d-48b1-8124-bfc190891886/api" unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.apicaller [9fa2c6] "unit-mysql-router-k8s-0" successfully connected to "10.152.183.186:17070" unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.logger logger worker started unit-mysql-router-k8s-0: 15:49:46 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" controller-0: 15:49:46 INFO juju.worker.raft.raftforwarder claiming lease "9fa2c6a8-d23d-48b1-8124-bfc190891886:application-leadership#mysql-router-k8s#" for "mysql-router-k8s/0" unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.leadership mysql-router-k8s/0 promoted to leadership of mysql-router-k8s unit-mysql-router-k8s-0: 15:49:46 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-router-k8s-0 unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.uniter unit "mysql-router-k8s/0" started unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.uniter resuming charm install unit-mysql-router-k8s-0: 15:49:46 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-router-k8s-7 from API server unit-mysql-router-k8s-0: 15:49:46 INFO juju.downloader downloading from ch:amd64/jammy/mysql-router-k8s-7 unit-mysql-router-k8s-0: 15:49:47 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-router-k8s-7") unit-mysql-router-k8s-0: 15:49:47 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-router-k8s-7") unit-mysql-router-k8s-0: 15:49:52 INFO juju.worker.uniter hooks are retried true unit-mysql-router-k8s-0: 15:49:52 INFO juju.worker.uniter found queued "install" hook unit-mysql-router-k8s-0: 15:49:53 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/install. unit-mysql-router-k8s-0: 15:49:54 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 15:49:54 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 15:49:54 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-router-k8s-0: 15:49:55 ERROR unit.mysql-router-k8s/0.juju-log Failed to create k8s service Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status resp.raise_for_status() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/httpx/_models.py", line 749, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/api/v1/namespaces/controller/pods/mysql-router-k8s-0' For more information check: https://httpstatuses.com/404 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 225, in _on_leader_elected self._create_service(f"{self.app.name}-read-only", 6447) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 94, in _create_service pod0 = client.get( File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/client.py", line 108, in get return self._client.request("get", res=res, name=name, namespace=namespace) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 245, in request return self.handle_response(method, resp, br) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response self.raise_for_status(resp) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status raise transform_exception(e) lightkube.core.exceptions.ApiError: pods "mysql-router-k8s-0" not found unit-mysql-router-k8s-0: 15:49:55 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 15:49:56 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 15:49:56 INFO juju.worker.uniter found queued "start" hook unit-mysql-router-k8s-0: 15:49:56 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/start. unit-mysql-router-k8s-0: 15:49:57 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) ```
carlcsaposs-canonical commented 1 year ago

Replicated on revision 6 with: juju deploy mysql-router-k8s --channel edge --trust --revision 6

carlcsaposs-canonical commented 1 year ago

Was using controller model; no errors when running juju add-model foo before juju deploy