search

canonical / grafana-agent-operator

https://charmhub.io/grafana-agent
Apache License 2.0
4 stars 11 forks source link

Charm fails to install from latest/edge, revision 20 #30

Closed facundofc closed 11 months ago

facundofc commented 1 year ago

Bug Description

The charm fails a fresh install from latest/edge channel, revision is 20 at the time of writing.

To Reproduce

juju deploy ubuntu juju deploy grafana-agent --channel=latest/edge # optionally --revision=20 juju relate ubuntu grafana-agent

charm goes into error state on install hook

Environment

Using juju with an openstack provider with Jammy VMs.

The same exact issue was observed while upgrading from revision 12 to 20 but running on juju with MaaS provider.

Relevant log output

unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install Traceback (most recent call last):
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 41, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 285, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from cryptography import x509
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from cryptography.x509 import certificate_transparency
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install ImportError: libssl.so.3: cannot open shared object file: No such file or directory
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install 
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install During handling of the above exception, another exception occurred:
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install 
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install Traceback (most recent call last):
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "./src/charm.py", line 20, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install   File "/var/lib/juju/agents/unit-grafana-agent-fresh-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 51, in <module>
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install     raise ImportError(
unit-grafana-agent-fresh-0: 14:17:21 WARNING unit.grafana-agent-fresh/0.install ImportError: charms.tls_certificates_interface.v2.tls_certificates is missing; please get it through charmcraft fetch-lib
unit-grafana-agent-fresh-0: 14:17:21 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-grafana-agent-fresh-0: 14:17:21 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-grafana-agent-fresh-0: 14:17:21 DEBUG juju.machinelock machine lock released for grafana-agent-fresh/0 uniter (run install hook)
unit-grafana-agent-fresh-0: 14:17:21 DEBUG juju.worker.uniter.operation lock released for grafana-agent-fresh/0
unit-grafana-agent-fresh-0: 14:17:21 INFO juju.worker.uniter awaiting error resolution for "install" hook

Additional context

No response

nishant-dash commented 12 months ago

I tried this on both focal and jammy (principals of ubuntu) and I see the same messages in logs but on the focal version, the install hook fails with that message

ubuntu-focal/0*           active    idle   3        10.5.2.243             
  grafana-agent-fresh/1*  error     idle            10.5.2.243             hook failed: "install"

Machine  State    Address     Inst id                               Series  AZ    Message
...
3        started  10.5.2.243  4b643e4a-bfc4-470b-a8c9-92870c883cac  focal   nova  ACTIVE

For jammy, the unit is in a blocked state

ubuntu/1*                 active    idle   2        10.5.0.202             
  grafana-agent/1*        blocked   idle            10.5.0.202             send-remote-write: off, logging-consumer: off, grafana-cloud-config: off

Machine  State    Address     Inst id                               Series  AZ    Message
2        started  10.5.0.202  2af6bb68-a085-4196-a7a7-52dd121c473b  jammy   nova  ACTIVE
...

charm versions I used (ignore the status since ive removed them from my test setup)

App                  Version  Status       Scale  Charm          Channel      Rev  Exposed  Message
grafana-agent                 blocked          1  grafana-agent  latest/edge   20  no       send-remote-write: off, logging-consumer: off, grafana-cloud-config: off
grafana-agent-fresh           error            1  grafana-agent  latest/edge   20  no       hook failed: "install"  
ubuntu               22.04    maintenance      1  ubuntu         stable        24  no       
ubuntu-focal         20.04    maintenance      1  ubuntu         stable        24  no
sed-i commented 12 months ago

The key line is

ImportError: libssl.so.3: cannot open shared object file: No such file or directory

Similar to https://github.com/canonical/grafana-agent-operator/issues/5.

sed-i commented 11 months ago

Hi @facundofc, @nishant-dash,

  1. The install error is probably because of focal/jammy mismatch between the primary and the subordinate. That's why the libssl.so.3 error.
  2. The send-remote-write: off, logging-consumer: off, grafana-cloud-config: off message is valid and correct: it warns you that key relations are missing.
$ charmcraft status grafana-agent

Track    Base                  Channel    Version    Revision                                                                                                                         
latest   ubuntu 20.04 (amd64)  stable     19         19                                                                                                                               
                               candidate  19         19                                                                                                                               
                               beta       19         19                                                                                                                               
                               edge       21         21                                                                                                                               
         ubuntu 22.04 (amd64)  stable     20         20                                                                                                                               
                               candidate  20         20                                                                                                                               
                               beta       20         20                                                                                                                               
                               edge       20         20

Here's how you can deploy jammy or focal correctly:

juju deploy --channel=edge ubuntu primary-focal --base ubuntu@20.04
juju deploy --channel=edge ubuntu primary-jammy --base ubuntu@22.04
juju deploy --channel=edge grafana-agent agent-jammy --base ubuntu@22.04
juju deploy --channel=edge grafana-agent agent-focal --base ubuntu@20.04
juju relate primary-focal agent-focal
juju relate primary-jammy agent-jammy
series: jammy
applications:
  agent-focal:
    charm: grafana-agent
    channel: edge
    revision: 21
    series: focal
  agent-jammy:
    charm: grafana-agent
    channel: edge
    revision: 20
  primary-focal:
    charm: ubuntu
    channel: edge
    revision: 24
    series: focal
    num_units: 1
    to:
    - "0"
    constraints: arch=amd64
    storage:
      block: loop,100M
      files: rootfs,100M
  primary-jammy:
    charm: ubuntu
    channel: edge
    revision: 24
    num_units: 1
    to:
    - "1"
    constraints: arch=amd64
    storage:
      block: loop,100M
      files: rootfs,100M
machines:
  "0":
    constraints: arch=amd64
    series: focal
  "1":
    constraints: arch=amd64
relations:
- - primary-focal:juju-info
  - agent-focal:juju-info
- - primary-jammy:juju-info
  - agent-jammy:juju-info
Model  Controller  Cloud/Region         Version  SLA          Timestamp
ga     lxd         localhost/localhost  3.1.6    unsupported  13:45:14-05:00

App            Version  Status   Scale  Charm          Channel  Rev  Exposed  Message
agent-focal             blocked      1  grafana-agent  edge      21  no       grafana-cloud-config: off, send-remote-write: off, logging-consumer: off
agent-jammy             blocked      1  grafana-agent  edge      20  no       logging-consumer: off, grafana-cloud-config: off, send-remote-write: off
primary-focal  20.04    active       1  ubuntu         edge      24  no    
primary-jammy  22.04    active       1  ubuntu         edge      24  no    

Unit              Workload  Agent  Machine  Public address  Ports  Message
primary-focal/0*  active    idle   0        10.41.168.157           
  agent-focal/0*  blocked   idle            10.41.168.157          grafana-cloud-config: off, send-remote-write: off, logging-consumer: off
primary-jammy/0*  active    idle   1        10.41.168.179           
  agent-jammy/0*  blocked   idle            10.41.168.179          logging-consumer: off, grafana-cloud-config: off, send-remote-write: off

Machine  State    Address        Inst id        Base          AZ  Message
0        started  10.41.168.157  juju-9cc8bd-0  ubuntu@20.04      Running
1        started  10.41.168.179  juju-9cc8bd-1  ubuntu@22.04      Running

Integration provider     Requirer               Interface              Type         Message
agent-focal:peers        agent-focal:peers      grafana_agent_replica  peer
agent-jammy:peers        agent-jammy:peers      grafana_agent_replica  peer
primary-focal:juju-info  agent-focal:juju-info  juju-info              subordinate
primary-jammy:juju-info  agent-jammy:juju-info  juju-info              subordinate

Potentially related:

Closing for now. Feel free to reopen if the issue is elsewhere.