github-actions[bot]
commented
1 year ago Open gboutry opened 1 year ago
github-actions[bot]
commented
1 year ago 
gboutry
commented
1 year ago Another run with debug logs: mysql2.log
relation data:
jhack show-relation mysql:database-peers --format json
{
"type": "peer",
"endpoints": [
{
"url": "mysql:database-peers",
"relation_id": 8,
"meta": {
"scale": 1,
"units": [
0
],
"leader_id": 0
},
"application_data": {
"cluster-name": "cluster-58a3017b136b545d62e8d3b9fb03c6eb",
"cluster-set-domain-name": "cluster-set-58a3017b136b545d62e8d3b9fb03c6eb",
"secret-id": "secret://4cad2829-f96b-45b8-8277-47786a891317/ck8o22698o8tjbgj5gtg"
},
"units_data": {
"mysql/0": {
"unit-status": "alive"
}
},
"model": null,
"other_model": null
}
]
}
paulomach
commented
1 year ago @gboutry thanks for the report. Did you saw that on other juju version?
gboutry
commented
1 year ago @gboutry thanks for the report. Did you saw that on other juju version?
I haven't, but I can try more
motjuste
commented
9 months ago I was using microk8s 1.24/stable with Juju 3.1.6, and kept facing this issue. Moving to microk8s 1.26/stable or 1.26-strict/stable worked.
paulomach
commented
9 months ago @motjuste , this is potentially related to issue #337 , which is fixed on 8.0/beta. Can you test with it?
motjuste
commented
9 months ago I have unfortunately moved to 1.26 microk8s ... will update if I manage to get something running somewhere.
gboutry
commented
8 months ago I have not seen this error for weeks now, seems fixed from my side.
motjuste
commented
8 months ago I have found something weird ... this problem occurs every time if I enable the mayastor add-on in microk8s 1.26 ... and simply enabling it causes this problem. Once I disable mayastor, the problem with the deployed MLFlow does not fix itself, but a fresh deployment after destroying the deployed apps of the MLFlow charm ends up working.
swasenius
commented
6 months ago Dear all I seem to have the same issue, but I don't know what mayastor is so hard to disable it. What else could possibly cause this?
$ juju status mysql-k8s
Model Controller Cloud/Region Version SLA Timestamp
kubeflow datalake-kubeflow-test-v3 datalake-kubeflow-test.k8s.local 3.4.0 unsupported 21:48:59+03:00
App Version Status Scale Charm Channel Rev Address Exposed Message
mysql-k8s waiting 1 mysql-k8s 8.0/stable 113 100.99.195.98 no waiting for units to settle down
Unit Workload Agent Address Ports Message
mysql-k8s/0* maintenance executing 100.120.219.253 installing charm softwareunit-katib-db-mysql-0: 21:48:19 ERROR unit.katib-db-mysql/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 786, in <module>
main(MySQLOperatorCharm)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/main.py", line 436, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 351, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 853, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/framework.py", line 942, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 583, in _on_mysql_pebble_ready
self._reconcile_pebble_layer(container)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/./src/charm.py", line 345, in _reconcile_pebble_layer
self._mysql.wait_until_mysql_connection()
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 379, in __call__
do = self.iter(retry_state=retry_state)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 325, in iter
raise retry_exc.reraise()
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 158, in reraise
raise self.last_attempt.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/tenacity/__init__.py", line 382, in __call__
result = fn(*args, **kwargs)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/src/mysql_k8s_helpers.py", line 236, in wait_until_mysql_connection
if check_port and not self.check_mysqlsh_connection():
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 2561, in check_mysqlsh_connection
self._run_mysqlsh_script("\n".join(connect_commands))
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/src/mysql_k8s_helpers.py", line 666, in _run_mysqlsh_script
stdout, _ = process.wait_output()
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1436, in wait_output
exit_code: int = self._wait()
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1375, in _wait
change = self._client.wait_change(self._change_id, timeout=timeout)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1893, in wait_change
return self._wait_change_using_wait(change_id, timeout)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1914, in _wait_change_using_wait
return self._wait_change(change_id, this_timeout)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1928, in _wait_change
resp = self._request('GET', f'/v1/changes/{change_id}/wait', query)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1655, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 1690, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/var/lib/juju/agents/unit-katib-db-mysql-0/charm/venv/ops/pebble.py", line 311, in http_open
return self.do_open(_UnixSocketConnection, req, # type:ignore
File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
r = h.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
unit-katib-db-mysql-0: 21:48:39 ERROR juju.worker.uniter.operation hook "mysql-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-katib-db-mysql-0: 21:48:39 ERROR juju.worker.uniter pebble poll failed for container "mysql": failed to send pebble-ready event: hook failed
Steps to reproduce
Expected behavior
Charm should become ready
Actual behavior
Charm has an unhandled exception and never becomes ready
Versions
Operating system: Ubuntu 22.04.3 LTS
Juju CLI: 3.2.3-genericlinux-amd64
Juju agent: 3.2.3
Charm revision: 99 (8.0/candidate)
microk8s: MicroK8s v1.26.8 revision 5883
Log output
Status log:
Additional context
I does not happen at every run, only some of them, but if it happens, mysql never become ready.