search

canonical / cassandra-k8s-operator

Apache License 2.0
1 stars 4 forks source link

Upgrade error when upgrading the edge version with a locally built charm from main #51

Open sed-i opened 2 years ago

sed-i commented 2 years ago

Describe the bug Upgrading from edge to latest source fails: the stop hook errors out.

To Reproduce

  1. juju deploy --channel=edge cassandra-k8s cass
  2. charmcraft pack
  3. curl -L https://github.com/instaclustr/cassandra-exporter/releases/download/v0.9.10/cassandra-exporter-agent-0.9.10.jar -o cassandra-exporter-agent.jar
  4. juju refresh ./cassandra-k8s_ubuntu-20.04-amd64.charm cass --resource cassandra-image='cassandra:3.11' --resource cassandra-prometheus-exporter="$(pwd)/cassandra-exporter-agent.jar"

Expected behavior Upgrade succeeds without an error.

Environment

Logs ``` unit-cass-0: 07:04:30.920 ERROR unit.cass/0.juju-log Uncaught exception while in charm code: Traceback (most recent call last): File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/pebble.py", line 67, in connect self.sock.connect(self.socket_path) FileNotFoundError: [Errno 2] No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/pebble.py", line 1142, in _request_raw response = self.opener.open(request, timeout=self.timeout) File "/usr/lib/python3.8/urllib/request.py", line 525, in open response = self._open(req, data) File "/usr/lib/python3.8/urllib/request.py", line 542, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(*args) File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/pebble.py", line 81, in http_open return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path) File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open raise URLError(err) urllib.error.URLError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./src/charm.py", line 297, in main(CassandraOperatorCharm, use_juju_for_storage=True) File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/main.py", line 424, in main framework.reemit() File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/framework.py", line 746, in reemit self._reemit() File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/framework.py", line 783, in _reemit custom_handler(event) File "./src/charm.py", line 77, in on_pebble_ready if self._set_config_file(event) is False: File "./src/charm.py", line 284, in _set_config_file if yaml.safe_load(self._container.pull(CONFIG_PATH).read()) != conf: File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/model.py", line 1219, in pull return self._pebble.pull(path, encoding=encoding) File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/pebble.py", line 1460, in pull response = self._request_raw('GET', '/v1/files', query, headers) File "/var/lib/juju/agents/unit-cass-0/charm/venv/ops/pebble.py", line 1155, in _request_raw raise ConnectionError(e.reason) ops.pebble.ConnectionError: [Errno 2] No such file or directory unit-cass-0: 07:04:31.171 ERROR juju.worker.uniter.operation hook "stop" (via hook dispatching script: dispatch) failed: exit status 1 unit-cass-0: 07:04:35.915 ERROR juju.worker.uniter pebble poll failed for container "cassandra": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/cassandra/pebble.socket: connect: no such file or directory ```
sed-i commented 2 years ago

This happens when upgrading the charm before pebble-ready. Here are more recent logs with more context.

unit-cassandra-0: 22:38:48.114 DEBUG unit.cassandra/0.juju-log Re-emitting <PebbleReadyEvent via CassandraOperatorCharm/on/cassandra_pebble_ready[13]>.
unit-cassandra-0: 22:38:48.144 ERROR unit.cassandra/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 70, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1315, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 84, in http_open
    return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 308, in <module>
    main(CassandraOperatorCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/main.py", line 429, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 753, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 88, in on_pebble_ready
    if self._set_config_file(event) is False:
  File "./src/charm.py", line 295, in _set_config_file
    if yaml.safe_load(self._container.pull(CONFIG_PATH).read()) != conf:
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/model.py", line 1257, in pull
    return self._pebble.pull(path, encoding=encoding)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1633, in pull
    response = self._request_raw('GET', '/v1/files', query, headers)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1328, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory
sed-i commented 2 years ago

Another flavor of an upgrade error:

unit-cassandra-0: 23:08:01 ERROR unit.cassandra/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 70, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1315, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 84, in http_open
    return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 308, in <module>
    main(CassandraOperatorCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 88, in on_pebble_ready
    if self._set_config_file(event) is False:
  File "./src/charm.py", line 295, in _set_config_file
    if yaml.safe_load(self._container.pull(CONFIG_PATH).read()) != conf:
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/model.py", line 1257, in pull
    return self._pebble.pull(path, encoding=encoding)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1633, in pull
    response = self._request_raw('GET', '/v1/files', query, headers)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1328, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory
unit-cassandra-0: 23:08:01 ERROR juju.worker.uniter.operation hook "cassandra-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-cassandra-0: 23:08:01 ERROR juju.worker.uniter pebble poll failed for container "cassandra": hook failed
unit-cassandra-0: 23:09:58 ERROR unit.cassandra/0.juju-log log-proxy:4: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 308, in <module>
    main(CassandraOperatorCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/lib/charms/loki_k8s/v0/loki_push_api.py", line 1761, in _on_relation_changed
    self._container.restart(WORKLOAD_SERVICE_NAME)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/model.py", line 1152, in restart
    self._pebble.restart_services(service_names)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1462, in restart_services
    return self._services_action('restart', services, timeout, delay)
  File "/var/lib/juju/agents/unit-cassandra-0/charm/venv/ops/pebble.py", line 1480, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "promtail" (cannot start service: exited quickly with code 1)
----- Logs from task 1 -----
2022-04-06T03:09:58Z INFO Most recent service output:
    (...)
    level=info ts=2022-04-06T03:09:29.877566465Z caller=tailer.go:142 component=tailer msg="tail routine: tail channel closed, stopping tailer" path=/var/log/cassandra/debug.log reason=null
    level=info ts=2022-04-06T03:09:29.877576614Z caller=tailer.go:135 component=tailer msg="tail routine: exited" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:29.877582184Z caller=tailer.go:212 component=tailer msg="stopped tailing file" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:29.877593483Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:29.937641841Z caller=server.go:260 http=[::]:9080 grpc=[::]:9095 msg="server listening on addresses"
    level=info ts=2022-04-06T03:09:29.937757297Z caller=main.go:119 msg="Starting Promtail" version="(version=2.4.1, branch=HEAD, revision=f61a4d261)"
    level=info ts=2022-04-06T03:09:31.544139849Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
    level=info ts=2022-04-06T03:09:31.581607171Z caller=server.go:260 http=[::]:9080 grpc=[::]:9095 msg="server listening on addresses"
    level=info ts=2022-04-06T03:09:31.581741536Z caller=main.go:119 msg="Starting Promtail" version="(version=2.4.1, branch=HEAD, revision=f61a4d261)"
    level=info ts=2022-04-06T03:09:36.581501511Z caller=filetargetmanager.go:255 msg="Adding target" key="/var/log/cassandra/debug.log:{job=\"juju_test-rerelate-byzo_c778cb16-514b-4456-825e-0bfd58ecc014_cassandra_cassandra-k8s\", juju_application=\"cassandra\", juju_charm=\"cassandra-k8s\", juju_model=\"test-rerelate-byzo\", juju_model_uuid=\"c778cb16-514b-4456-825e-0bfd58ecc014\", juju_unit=\"cassandra/0\"}"
    level=info ts=2022-04-06T03:09:36.581827786Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/cassandra/debug.log
    ts=2022-04-06T03:09:36.581876514Z caller=log.go:168 level=info msg="Seeked /var/log/cassandra/debug.log - &{Offset:680350 Whence:0}"
    level=info ts=2022-04-06T03:09:58.519603887Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
    level=info ts=2022-04-06T03:09:58.524743099Z caller=filetargetmanager.go:316 msg="Removing target" key="/var/log/cassandra/debug.log:{job=\"juju_test-rerelate-byzo_c778cb16-514b-4456-825e-0bfd58ecc014_cassandra_cassandra-k8s\", juju_application=\"cassandra\", juju_charm=\"cassandra-k8s\", juju_model=\"test-rerelate-byzo\", juju_model_uuid=\"c778cb16-514b-4456-825e-0bfd58ecc014\", juju_unit=\"cassandra/0\"}"
    level=info ts=2022-04-06T03:09:58.541530184Z caller=tailer.go:99 component=tailer msg="position timer: exited" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:58.541575723Z caller=tailer.go:142 component=tailer msg="tail routine: tail channel closed, stopping tailer" path=/var/log/cassandra/debug.log reason=null
    level=info ts=2022-04-06T03:09:58.541588042Z caller=tailer.go:135 component=tailer msg="tail routine: exited" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:58.541593632Z caller=tailer.go:212 component=tailer msg="stopped tailing file" path=/var/log/cassandra/debug.log
    level=info ts=2022-04-06T03:09:58.541611511Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/cassandra/debug.log
    level=error ts=2022-04-06T03:09:58.577364581Z caller=main.go:115 msg="error creating promtail" error="at least one client config should be provided"
2022-04-06T03:09:58Z ERROR cannot start service: exited quickly with code 1
-----
unit-cassandra-0: 23:09:58 ERROR juju.worker.uniter.operation hook "log-proxy-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1