search

addyess / charm-splunk-k8s

Juju Charm Operator for managing the splunk app in kubernetes
Apache License 2.0
0 stars 0 forks source link

pebble poll failed for container "splunk" #3

Closed dasm closed 3 years ago

dasm commented 3 years ago

When deployed from local build, the application waits for accept-license command. After running it, it doesn't progress, but in logs can be found

unit-splunk-k8s-0: 09:48:44 ERROR juju.worker.uniter pebble poll failed for container "splunk": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/splunk/pebble.socket: connect: no such file or directory
unit-splunk-k8s-0: 09:48:45 ERROR unit.splunk-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 58, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 729, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 72, in http_open
    return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
  File "/usr/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 186, in <module>
    main(SplunkCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/main.py", line 406, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 93, in _on_accept_license_action
    self._do_config_change()
  File "./src/charm.py", line 140, in _do_config_change
    services = container.get_plan().to_dict().get("services", {})
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/model.py", line 1071, in get_plan
    return self._pebble.get_plan()
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 883, in get_plan
    resp = self._request('GET', '/v1/plan', {'format': 'yaml'})
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 700, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 742, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory
jameinel commented 3 years ago

If you are calling accept_license_action before pebble_ready has fired, then it is plausible the sidecar where pebble resides does not exist yet.

addyess commented 3 years ago

@dasm -- i don't think the charm is even looking for pebble_ready event. I suppose it's possible this event was run before the pebble ready.

dasm commented 3 years ago

Yes, seems like it was the case @jameinel I reproduced it by:

# terminal 1
juju deploy ./splunk-k8s.charm --resource splunk-image=splunk/splunk:latest

# terminal 2 waiting to see when app is ready to "accept-license"
watch juju status

# terminal 3
juju run-action splunk-k8s/0 accept-license

# terminal 4
juju debug-log

When I ran accept-license first time, I might've done it before pebble was configured, because I could see 

unit-splunk-k8s-0: 12:07:57 ERROR unit.splunk-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 58, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 729, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 72, in http_open
    return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
  File "/usr/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 186, in <module>
    main(SplunkCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/main.py", line 406, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 93, in _on_accept_license_action
    self._do_config_change()
  File "./src/charm.py", line 140, in _do_config_change
    services = container.get_plan().to_dict().get("services", {})
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/model.py", line 1071, in get_plan
    return self._pebble.get_plan()
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 883, in get_plan
    resp = self._request('GET', '/v1/plan', {'format': 'yaml'})
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 700, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-splunk-k8s-0/charm/venv/ops/pebble.py", line 742, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory

However, after rerunning accept-license in few seconds, it went through and it worked.

I think this one is invalid.

Thanks @addyess and @jameinel