search

canonical / kfp-operators

Kubeflow Pipelines Operators
Apache License 2.0
2 stars 12 forks source link

`kfp-api` integration test fails with `mysql-k8s` unit in error #377

Closed NohaIhab closed 9 months ago

NohaIhab commented 1 year ago

Bug Description

This is an issue against branch 1.8-updates-dev-branch the test test_relational_db_relation_with_mysql_relation fails intermittently with the mysql-k8s unit in error as seen for example in this run

To Reproduce

run the kfp-api integration tests in the CI on branch 1.8-updates-dev-branch

Environment

microk8s 1.25-strict/stable juju 3.1/stable

Relevant log output

INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model            Controller                Cloud/Region        Version  SLA          Timestamp
test-charm-gwks  github-pr-2798a-microk8s  microk8s/localhost  3.1.6    unsupported  10:05:05Z

App        Version                Status   Scale  Charm                    Channel         Rev  Address         Exposed  Message
kfp-api                           active       1  kfp-api                                    0  10.152.183.42   no       
kfp-db     mariadb/server:10.3    active       1  charmed-osm-mariadb-k8s  latest/stable    35  10.152.183.201  no       ready
kfp-viz    res:oci-image@ffaf37e  active       1  kfp-viz                  2.0/stable      822  10.152.183.135  no       
minio      res:oci-image@1755999  active       1  minio                    ckf-1.7/stable  214  10.152.183.216  no       
mysql-k8s                         waiting      1  mysql-k8s                8.0/edge        110  10.152.183.147  no       installing agent

Unit          Workload  Agent  Address       Ports          Message
kfp-api/0*    active    idle   10.1.106.201                 
kfp-db/0*     active    idle   10.1.106.207  3306/TCP       ready
kfp-viz/0*    active    idle   10.1.106.209  8888/TCP       
minio/0*      active    idle   10.1.106.210  9000-9001/TCP  
mysql-k8s/0*  error     idle   10.1.106.212                 hook failed: "config-changed"

INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:

unit-kfp-api-0: 10:00:32 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 10:00:32 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[21]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:00:36 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 10:00:37 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent via KfpApiOperator/on/apiserver_pebble_ready[26]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:00:42 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 10:00:42 ERROR unit.kfp-api/0.juju-log Failed to handle <ConfigChangedEvent via KfpApiOperator/on/config_changed[31]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:00:52 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to generate container configuration.
unit-kfp-api-0: 10:00:52 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationJoinedEvent via KfpApiOperator/on/mysql_relation_joined[41]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:01:02 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to generate container configuration.
unit-kfp-api-0: 10:01:02 ERROR unit.kfp-api/0.juju-log mysql:0: Failed to handle <RelationChangedEvent via KfpApiOperator/on/mysql_relation_changed[46]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:01:13 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to generate container configuration.
unit-kfp-api-0: 10:01:13 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to handle <RelationChangedEvent via KfpApiOperator/on/object_storage_relation_changed[56]> with error: List of <ops.model.Relation object-storage:1> versions not found for apps: minio
unit-kfp-api-0: 10:01:21 ERROR unit.kfp-api/0.juju-log kfp-viz:2: Failed to generate container configuration.
unit-kfp-api-0: 10:01:21 ERROR unit.kfp-api/0.juju-log kfp-viz:2: Failed to handle <RelationChangedEvent via KfpApiOperator/on/kfp_viz_relation_changed[61]> with error: Waiting for kfp-viz relation data
unit-kfp-api-0: 10:01:28 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to generate container configuration.
unit-kfp-api-0: 10:01:28 ERROR unit.kfp-api/0.juju-log object-storage:1: Failed to handle <RelationChangedEvent via KfpApiOperator/on/object_storage_relation_changed[66]> with error: Waiting for kfp-viz relation data
unit-mysql-k8s-0: 10:04:59 ERROR unit.mysql-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 272, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1599, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 286, in http_open
    return self.do_open(_UnixSocketConnection, req,  # type:ignore
  File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/./src/charm.py", line 786, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/./src/charm.py", line 412, in _on_config_changed
    if not self._mysql.is_mysqld_running():
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 564, in is_mysqld_running
    return self.container.exists(MYSQLD_SOCK_FILE)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/model.py", line 2444, in exists
    self._pebble.list_files(str(path), itself=True)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 2104, in list_files
    resp = self._request('GET', '/v1/files', query)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1564, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1612, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory
unit-mysql-k8s-0: 10:05:00 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response

orfeas-k commented 9 months ago

kfp-peristence charm integration tests also fail wit the same error https://github.com/canonical/pipelines-rocks/actions/runs/7528345532/job/20493131512#step:9:77

syncronize-issues-to-jira[bot] commented 9 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5203.

This message was autogenerated

orfeas-k commented 9 months ago

This was also observed in bundle-integration-v* tests which also deploy mysql-k8s from 8.0/stable.