search

canonical / sysbench-operator

Charm to manage sysbench and connect with Data Platform databases
Apache License 2.0
1 stars 0 forks source link

Cannot create cross-controller relation sysbench/lxd->postgresql/microk8s: ERROR creating remote secret client: cannot resolve "controller-service.controller-m2k8s.svc.cluster.local"... i/o timeout #38

Open taurus-forever opened 2 months ago

taurus-forever commented 2 months ago

Steps to reproduce

  1. bootstrap lxd and microk8s controllers on the same client
  2. deploy posrtgresql-k8s on microk8s (14/edge)
  3. deploy sysbench on lxd
  4. juju offer posrtgresql-k8s:database # on microk8s
  5. juju consume microk8s:admin/mymodel.postgresql-k8s
  6. juju relate sysbench postgresql-k8s

Expected behavior

Succesful relation to workaround https://github.com/canonical/sysbench-operator/issues/37

Actual behavior

ubuntu@juju340:~$ controller-0: 08:30:47 INFO juju.worker.firewaller start "sysbench:postgresql postgresql-k8s:database"
unit-sysbench-0: 08:30:48 INFO juju.worker.uniter.operation ran "postgresql-relation-created" hook (via hook dispatching script: dispatch)
unit-sysbench-0: 08:30:48 INFO juju.worker.uniter.operation ran "postgresql-relation-joined" hook (via hook dispatching script: dispatch)
unit-sysbench-0: 08:30:49 INFO juju.worker.uniter.operation ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
controller-0: 08:30:52 INFO juju.worker.firewaller stopped "sysbench:postgresql postgresql-k8s:database", err: cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 INFO juju.worker.firewaller non-fatal error "sysbench:postgresql postgresql-k8s:database": cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 ERROR juju.worker.firewaller exited "sysbench:postgresql postgresql-k8s:database": cannot open facade to remote model to publish network change: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
controller-0: 08:30:52 INFO juju.worker.firewaller restarting "sysbench:postgresql postgresql-k8s:database" in 1m0s
unit-sysbench-0: 08:30:52 ERROR unit.sysbench/0.juju-log postgresql:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3022, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-sysbench-0/secret-get', 'secret://77fe3914-0206-4d12-8735-32ebe4c82a49/corif6fmp25c760hv220', '--label', 'postgresql.1.user.secret', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/./src/charm.py", line 325, in <module>
    main(SysbenchOperator)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2846, in _on_relation_changed_event
    self.relation_data._register_secrets_to_relation(event.relation, diff.added)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1604, in _register_secrets_to_relation
    self._register_secret_to_relation(
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1590, in _register_secret_to_relation
    CachedSecret(self._model, self.component, label, secret_id).meta
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 280, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3370, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3024, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR creating remote secret client: cannot resolve "controller-service.controller-mk8s.svc.cluster.local": lookup controller-service.controller-mk8s.svc.cluster.local: i/o timeout
...

unit-sysbench-0: 08:31:06 ERROR juju.worker.uniter.operation hook "postgresql-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-sysbench-0: 08:31:06 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-sysbench-0: 08:31:16 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-sysbench-0: 08:31:16 ERROR unit.sysbench/0.juju-log postgresql:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3022, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-sysbench-0/secret-get', 'secret://77fe3914-0206-4d12-8735-32ebe4c82a49/corif6fmp25c760hv220', '--label', 'postgresql.1.user.secret', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-sysbench-0/charm/./src/charm.py", line 325, in <module>
    main(SysbenchOperator)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2846, in _on_relation_changed_event
    self.relation_data._register_secrets_to_relation(event.relation, diff.added)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1604, in _register_secrets_to_relation
    self._register_secret_to_relation(
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1590, in _register_secret_to_relation
    CachedSecret(self._model, self.component, label, secret_id).meta
  File "/var/lib/juju/agents/unit-sysbench-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 280, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3370, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-sysbench-0/charm/venv/ops/model.py", line 3024, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR getting cluster client: unable to determine legacy status for namespace "test59": Get "https://127.0.0.1:16443/api/v1/namespaces/test59": dial tcp 127.0.0.1:16443: connect: connection refused

unit-sysbench-0: 08:31:16 ERROR juju.worker.uniter.operation hook "postgresql-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-sysbench-0: 08:31:16 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook

Versions

Operating system: Ubuntu 23.10 Juju CLI: 3.4.2-genericlinux-arm64 Juju agent: see above Charm revision: see above LXD: 5.21.1-2d13beb rev28463 MicroK8s: MicroK8s v1.30.0 revision 6788 (strict)

Juju status:

ubuntu@juju340:~$ juju status
Model   Controller  Cloud/Region         Version  SLA          Timestamp
test61  lxd         localhost/localhost  3.4.2    unsupported  08:32:25+02:00

SAAS            Status  Store  URL
postgresql-k8s  active  mk8s   admin/test59.postgresql-k8s

App       Version  Status  Scale  Charm     Channel        Rev  Exposed  Message
sysbench           error       1  sysbench  latest/stable   26  no       hook failed: "postgresql-relation-changed"

Unit         Workload  Agent  Machine  Public address  Ports  Message
sysbench/0*  error     idle   0        10.184.219.31          hook failed: "postgresql-relation-changed"

Machine  State    Address        Inst id        Base          AZ  Message
0        started  10.184.219.31  juju-9e6ab4-0  ubuntu@22.04      Running
ubuntu@juju340:~$ juju status -m mk8s:test59
Model   Controller  Cloud/Region        Version  SLA          Timestamp
test59  mk8s        microk8s/localhost  3.4.2    unsupported  08:32:38+02:00

App                  Version  Status  Scale  Charm                Channel        Rev  Address         Exposed  Message
postgresql-k8s       14.11    active      1  postgresql-k8s       14/edge        247  10.152.183.162  no       

Unit                    Workload  Agent  Address       Ports  Message
postgresql-k8s/0*       active    idle   10.1.204.230         Primary

Offer           Application     Charm           Rev  Connected  Endpoint  Interface          Role
postgresql-k8s  postgresql-k8s  postgresql-k8s  247  1/1        database  postgresql_client  provider
ubuntu@juju340:~$
github-actions[bot] commented 2 months ago

https://warthogs.atlassian.net/browse/DPE-4288