search

canonical / grafana-agent-operator

https://charmhub.io/grafana-agent
Apache License 2.0
4 stars 11 forks source link

rev 234 of `grafana-agent-operator` fails at install #165

Closed phvalguima closed 3 months ago

phvalguima commented 3 months ago

Bug Description

working: rev 223, executed 1 day away - CI run example broken: rev 234, executed 7 hours ago - CI run example

There was no changes in this matter from our side.

Seems on the newest release, the new revision is having problems importing cryptography.

To Reproduce

juju add-model test juju deploy self-signed-certificates juju deploy ubuntu juju deploy grafana-agent --channel=latest/edge juju relate ubuntu grafana-agent

Environment

Juju 3.4.5 grafana-agent 234

Relevant log output

Test above:

$ juju status --relations
Model  Controller           Cloud/Region         Version  SLA          Timestamp
test   localhost-localhost  localhost/localhost  3.4.5    unsupported  11:07:17+02:00

App                       Version  Status  Scale  Charm                     Channel        Rev  Exposed  Message
grafana-agent                      error       1  grafana-agent             latest/edge    234  no       hook failed: "install"
self-signed-certificates           active      1  self-signed-certificates  latest/stable  155  no       
ubuntu                    22.04    active      1  ubuntu                    latest/stable   24  no       

Unit                         Workload  Agent  Machine  Public address  Ports  Message
self-signed-certificates/0*  active    idle   0        10.115.236.11          
ubuntu/0*                    active    idle   1        10.115.236.246         
  grafana-agent/0*           error     idle            10.115.236.246         hook failed: "install"

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.115.236.11   juju-963ac2-0  ubuntu@22.04      Running
1        started  10.115.236.246  juju-963ac2-1  ubuntu@22.04      Running

Integration provider  Requirer                 Interface              Type         Message
grafana-agent:peers   grafana-agent:peers      grafana_agent_replica  peer         joining  
ubuntu:juju-info      grafana-agent:juju-info  juju-info              subordinate

From one of the CIs, the grafana-agent seems to be failing early, when trying to load its dependencies:

unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from cryptography import x509
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install 
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install The above exception was the direct cause of the following exception:
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install 
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install Traceback (most recent call last):
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install   File "/var/lib/juju/agents/unit-grafana-agent-2/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install     raise ImportError(
unit-grafana-agent-2: 23:28:02 WARNING unit.grafana-agent/2.install ImportError: failed to import charms.tls_certificates_interface.v2.tls_certificates; Either the library itself is missing (please get it through charmcraft fetch-lib) or one of its dependencies is unmet.
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install Traceback (most recent call last):
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from cryptography import x509
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install 
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install The above exception was the direct cause of the following exception:
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install 
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install Traceback (most recent call last):
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install   File "/var/lib/juju/agents/unit-grafana-agent-3/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install     raise ImportError(
unit-grafana-agent-3: 23:28:02 WARNING unit.grafana-agent/3.install ImportError: failed to import charms.tls_certificates_interface.v2.tls_certificates; Either the library itself is missing (please get it through charmcraft fetch-lib) or one of its dependencies is unmet.
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install Traceback (most recent call last):
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from cryptography import x509
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install 
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install The above exception was the direct cause of the following exception:
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install 
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install Traceback (most recent call last):
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install   File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-1: 23:28:02 WARNING unit.grafana-agent/1.install     raise ImportError(


### Additional context

_No response_
mmkay commented 3 months ago

What changed in between: releasing #160 to cater for an issue with newer charmcraft version.

mmkay commented 3 months ago

@phvalguima, could you please check if grafana-agent charm is the same series as the principal? This was happening for series mismatches before.

taurus-forever commented 3 months ago

Principal is ubuntu 22.04 grafana-agent is 234 which is the same for 20.04 and 22.04: Screenshot from 2024-08-08 12-54-17 Screenshot from 2024-08-08 12-54-23

Maybe related to https://github.com/canonical/charmcraft/issues/1783 ?

phvalguima commented 3 months ago

@mmkay no luck

@phvalguima, could you please check if grafana-agent charm is the same series as the principal? This was happening for series mismatches before.

Testing with:

juju add-model test
juju deploy ubuntu --series=jammy
juju deploy grafana-agent --channel=latest/edge --series=jammy
juju relate ubuntu grafana-agent

Status:

$ juju status -m test --relations
Model  Controller           Cloud/Region         Version  SLA          Timestamp
test   localhost-localhost  localhost/localhost  3.4.5    unsupported  14:09:20+02:00

App            Version  Status  Scale  Charm          Channel        Rev  Exposed  Message
grafana-agent           error       1  grafana-agent  latest/edge    234  no       hook failed: "install"
ubuntu         22.04    active      1  ubuntu         latest/stable   24  no       

Unit                Workload  Agent  Machine  Public address  Ports  Message
ubuntu/0*           active    idle   0        10.115.236.229         
  grafana-agent/0*  error     idle            10.115.236.229         hook failed: "install"

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.115.236.229  juju-bce162-0  ubuntu@22.04      Running

Integration provider  Requirer                 Interface              Type         Message
grafana-agent:peers   grafana-agent:peers      grafana_agent_replica  peer         joining  
ubuntu:juju-info      grafana-agent:juju-info  juju-info              subordinate

More details in status's JSON: https://pastebin.ubuntu.com/p/qHTR6kv3Cp/

Same thing:

unit-grafana-agent-0: 14:07:31 INFO juju.worker.uniter found queued "install" hook
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from cryptography import x509
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install The above exception was the direct cause of the following exception:
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install     raise ImportError(
unit-grafana-agent-0: 14:07:33 WARNING unit.grafana-agent/0.install ImportError: failed to import charms.tls_certificates_interface.v2.tls_certificates; Either the library itself is missing (please get it through charmcraft fetch-lib) or one of its dependencies is unmet.
unit-grafana-agent-0: 14:07:34 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-grafana-agent-0: 14:07:34 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:07:39 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from cryptography import x509
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install The above exception was the direct cause of the following exception:
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install     raise ImportError(
unit-grafana-agent-0: 14:07:39 WARNING unit.grafana-agent/0.install ImportError: failed to import charms.tls_certificates_interface.v2.tls_certificates; Either the library itself is missing (please get it through charmcraft fetch-lib) or one of its dependencies is unmet.
unit-grafana-agent-0: 14:07:40 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-grafana-agent-0: 14:07:40 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:07:50 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from cryptography import x509
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install The above exception was the direct cause of the following exception:
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install     raise ImportError(
unit-grafana-agent-0: 14:07:51 WARNING unit.grafana-agent/0.install ImportError: failed to import charms.tls_certificates_interface.v2.tls_certificates; Either the library itself is missing (please get it through charmcraft fetch-lib) or one of its dependencies is unmet.
unit-grafana-agent-0: 14:07:51 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-grafana-agent-0: 14:07:51 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:08:11 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 43, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from charms.tls_certificates_interface.v2.tls_certificates import (  # type: ignore
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/tls_certificates_interface/v2/tls_certificates.py", line 284, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from cryptography import x509
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/__init__.py", line 7, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from cryptography.x509 import certificate_transparency, verification
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/venv/cryptography/x509/certificate_transparency.py", line 11, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location)
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install The above exception was the direct cause of the following exception:
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install 
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install Traceback (most recent call last):
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/./src/charm.py", line 21, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from grafana_agent import METRICS_RULES_SRC_PATH, GrafanaAgentCharm
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/src/grafana_agent.py", line 31, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     from charms.observability_libs.v0.cert_handler import CertHandler
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install   File "/var/lib/juju/agents/unit-grafana-agent-0/charm/lib/charms/observability_libs/v0/cert_handler.py", line 53, in <module>
unit-grafana-agent-0: 14:08:12 WARNING unit.grafana-agent/0.install     raise ImportError(
mmkay commented 3 months ago

@phvalguima: yes, we think this is related to the issue in charmcraft @taurus-forever reported above. We're trying to get charmcraft to update stable to the version with the fix to re-publish affected track.

phvalguima commented 3 months ago

@mmkay ack. What I don't get is where the disconnect of charmhub.io having rev 234 is coming from.

Your latest merged commit is publishing rev 233 and it ran successfully its integration tests: https://github.com/canonical/grafana-agent-operator/actions/runs/10293632034/job/28492652548

Was 234 published manually after that?

mmkay commented 3 months ago

@phvalguima

No, rev234 was also published: https://github.com/canonical/grafana-agent-operator/actions/runs/10293632034/job/28492652437

There are four jobs that publish revisions of gagent for different bases and architectures. All of them usually publish a separate revision that's only valid for the specific base / architecture combination. This time it seems all of them act as if they are tailored for all bases and architectures.

phvalguima commented 3 months ago

No, rev234 was also published: https://github.com/canonical/grafana-agent-operator/actions/runs/10293632034/job/28492652437

@mmkay that CI is releasing rev 234 only for ARM. AMD releases are rev. 231 and 232 respectively.

Now, I've downloaded both arm64 and amd64 from charmhub:

$ juju download grafana-agent --channel=latest/edge --series=jammy --arch=arm64
WARNING series flag is deprecated, use --base instead
Fetching charm "grafana-agent" revision 234 using "latest/edge" channel and base "arm64/ubuntu/22.04"
Install the "grafana-agent" charm with:
    juju deploy ./grafana-agent_r234.charm
$ mv grafana-agent_arm64_r234.charm 
$ md5sum grafana-agent_arm64_r234.charm 
d350632f8af728af3b3b4dc6a82da6c7  grafana-agent_arm64_r234.charm

$ juju download grafana-agent --channel=latest/edge --series=jammy --arch=amd64
WARNING series flag is deprecated, use --base instead
Fetching charm "grafana-agent" revision 234 using "latest/edge" channel and base "amd64/ubuntu/22.04"
Install the "grafana-agent" charm with:
    juju deploy ./grafana-agent_r234.charm
$ md5sum grafana-agent_r234.charm 
d350632f8af728af3b3b4dc6a82da6c7  grafana-agent_r234.charm

If that is the case, then I am using an arm64 release on my amd64. That can explain why my CI fails whereas yours pass: you are pulling the code, building and then pushing to charmhub.

From @taurus-forever bug:

Example for this is the postgresql-tes-app packing and CI tests passed fine, using a locally packed charm, but both amd64 and arm64 revisions got published as amd64 revisions.

So, that is indeed the root cause.

ca-scribner commented 3 months ago

There's a new edge release here that is using charmcraft 3.1.2. Once that finishes (maybe an hour or so), we should check charmcraft status grafana-agent to confirm 3.1.2 fixes the issue.

mmkay commented 3 months ago

Based on charmhub, we're now back to having different revisions for different bases and architectures. Fetching charms confirms that too:

ubuntu@charm-dev:~$ juju download grafana-agent --channel=latest/edge --series=jammy --arch=arm64
WARNING series flag is deprecated, use --base instead
Fetching charm "grafana-agent" revision 239 using "latest/edge" channel and base "arm64/ubuntu/22.04"
Install the "grafana-agent" charm with:
    juju deploy ./grafana-agent_r239.charm
ubuntu@charm-dev:~$ md5sum grafana-agent_r239.charm 
202d84836da91878eb51c90cdee82c98  grafana-agent_r239.charm
ubuntu@charm-dev:~$ juju download grafana-agent --channel=latest/edge --series=jammy --arch=amd64
WARNING series flag is deprecated, use --base instead
Fetching charm "grafana-agent" revision 238 using "latest/edge" channel and base "amd64/ubuntu/22.04"
Install the "grafana-agent" charm with:
    juju deploy ./grafana-agent_r238.charm
ubuntu@charm-dev:~$ md5sum grafana-agent_r238.charm 
3fef6890aea0bea771c2377fef5c33c1  grafana-agent_r238.charm

Fetching latest/edge grafana-agent should be fine now.

phvalguima commented 3 months ago

nice, thanks @mmkay @ca-scribner @taurus-forever!