search

canonical / grafana-agent-operator

https://charmhub.io/grafana-agent
Apache License 2.0
4 stars 10 forks source link

Traces do not get correctly exported to tempo when mysql-router is related to grafana-agent #195

Open shayancanonical opened 2 days ago

shayancanonical commented 2 days ago

Bug Description

Since mysql-router is a subordinate charm, we first relate grafana-agent with the principal charm (on the juju-info interface). Then, we relate grafana-agent:cos-agent mysql-router. When we do so, I can see loki logs trickling through to grafana/loki. However, I am unable to see traces from mysql-router showing up in tempo.

Upon further inspection, I noticed an error trace in the debug-logs, which I have included below

To Reproduce

  1. checkout branch feature/tempo_ha in mysql-router (https://github.com/canonical/mysql-router-operator/tree/feature/tempo_ha)
  2. charmcraft pack
  3. juju add-model database (lxd controller)
  4. juju deploy mysql --channel 8.0/edge
  5. juju deploy ./
  6. juju deploy mysql-test-app
  7. juju deploy grafana-agent
  8. juju relate mysql mysql-router
  9. juju relate mysql-router mysql-test-app
  10. juju relate mysql-test-app:juju-info grafana-agent
  11. juju relate mysql-router:cos-agent grafana-agent
  12. <consume grafana, loki, prometheus and tracing offers from the cos model>
  13. juju relate grafana-agent grafana
  14. juju relate grafana-agent prometheus
  15. juju relate grafana-agent loki
  16. juju relate grafana-agent tempo

Environment

Juju: 3.5.4 LXD: 6.1

Model     Controller  Cloud/Region         Version  SLA          Timestamp
database  lxd-3-5-4   localhost/localhost  3.5.4    unsupported  19:22:17Z

SAAS        Status  Store       URL
grafana     active  uk8s-3-5-4  admin/cos.grafana
loki        active  uk8s-3-5-4  admin/cos.loki
prometheus  active  uk8s-3-5-4  admin/cos.prometheus
tempo       active  uk8s-3-5-4  admin/cos.tempo

App             Version          Status  Scale  Charm           Channel        Rev  Exposed  Message
grafana-agent                    active      1  grafana-agent   latest/stable  216  no       
mysql           8.0.37-0ubun...  active      1  mysql           8.0/edge       301  no       
mysql-router    8.0.37-0ubun...  active      1  mysql-router                     0  no       
mysql-test-app  0.0.2            active      1  mysql-test-app  latest/stable   51  no       Last written value=213191

Unit                Workload  Agent  Machine  Public address  Ports           Message
mysql-test-app/0*   active    idle   1        10.205.193.72                   Last written value=213191
  grafana-agent/0*  active    idle            10.205.193.72                   
  mysql-router/0*   active    idle            10.205.193.72                   
mysql/0*            active    idle   0        10.205.193.151  3306,33060/tcp  Primary

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.205.193.151  juju-4a96ba-0  ubuntu@22.04      Running
1        started  10.205.193.72   juju-4a96ba-1  ubuntu@22.04      Running

Integration provider                           Requirer                                       Interface                    Type         Message
grafana-agent:grafana-dashboards-provider      grafana:grafana-dashboard                      grafana_dashboard            regular      
grafana-agent:peers                            grafana-agent:peers                            grafana_agent_replica        peer         
loki:logging                                   grafana-agent:logging-consumer                 loki_push_api                regular      
mysql-router:cos                               mysql-router:cos                               cos                          peer         
mysql-router:cos-agent                         grafana-agent:cos-agent                        cos_agent                    subordinate  
mysql-router:database                          mysql-test-app:database                        mysql_client                 subordinate  
mysql-router:deprecated-shared-db-credentials  mysql-router:deprecated-shared-db-credentials  _deprecated_shared_db_peers  peer         
mysql-router:tls                               mysql-router:tls                               tls                          peer         
mysql-router:upgrade-version-a                 mysql-router:upgrade-version-a                 upgrade                      peer         
mysql-test-app:application-peers               mysql-test-app:application-peers               application-peers            peer         
mysql-test-app:juju-info                       grafana-agent:juju-info                        juju-info                    subordinate  
mysql:database                                 mysql-router:backend-database                  mysql_client                 regular      
mysql:database-peers                           mysql:database-peers                           mysql_peers                  peer         
mysql:restart                                  mysql:restart                                  rolling_op                   peer         
mysql:upgrade                                  mysql:upgrade                                  upgrade                      peer         
prometheus:receive-remote-write                grafana-agent:send-remote-write                prometheus_remote_write      regular      
tempo:tracing                                  grafana-agent:tracing                          tracing                      regular      

Storage Unit  Storage ID  Type        Pool    Mountpoint                      Size     Status    Message
mysql/0       database/0  filesystem  rootfs  /var/snap/charmed-mysql/common  1.8 TiB  attached

Relevant log output

unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log ops 2.14.0 up and running.
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Setting up span exporter to endpoint: http://10.0.0.44:4318/v1/traces
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Starting root trace with id='a1fe6a71459db2c991922d4c84bc0341'.                                          
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Emitting Juju event update_status.                                                                                                              
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Starting new HTTP connection (1): 10.0.0.44:4318
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log http://10.0.0.44:4318 "POST /v1/traces HTTP/11" 200 2
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log ops 2.9.0 up and running.                                                                                                                         
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log data={}                                                                                                                                           
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log failed to validate databag: {'egress-subnets': '10.205.193.72/32', 'ingress-address': '10.205.193.72', 'private-address': '10.205.193.72'}        
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/lib/charms/grafana_agent/v0/cos_agent.py", line 369, in load                                                                                    
    return cls.parse_raw(json.dumps(data))  # type: ignore                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 549, in parse_raw                                                                                                              
    return cls.parse_obj(obj)                                                                                                                                                                                      
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 526, in parse_obj                                                                                                              
    return cls(**obj)                                                                                    
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 341, in __init__                                                                                                               
    raise validation_error                                                                               
pydantic.error_wrappers.ValidationError: 1 validation error for CosAgentRequirerUnitData                 
receivers                                                                                                
  field required (type=value_error.missing)                                                              
unit-mysql-router-0: 19:04:27 INFO unit.mysql-router/0.juju-log failed validating relation data for <ops.model.Relation cos-agent:16>                                                                  
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log Emitting Juju event update_status.

Additional context

No response

shayancanonical commented 2 days ago

Confirmed that the issue exists with latest/edge of grafana-agent too:

unit-grafana-agent-1: 20:03:56 DEBUG unit.grafana-agent/1.juju-log tracing:46: failed to validate databag: {}
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/tempo_k8s/v2/tracing.py", line 209, in load
    return cls.parse_raw(json.dumps(data))  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 341, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 1 validation error for TracingProviderAppData
receivers
  field required (type=value_error.missing)
unit-grafana-agent-1: 20:03:56 INFO unit.grafana-agent/1.juju-log tracing:46: failed validating relation data for <ops.model.Relation tracing:46>
unit-grafana-agent-1: 20:03:56 DEBUG unit.grafana-agent/1.juju-log tracing:46: Emitting Juju event tracing_relation_created.
unit-grafana-agent-1: 20:03:57 WARNING unit.grafana-agent/1.tracing-relation-joined /var/lib/juju/agents/unit-grafana-agent-1/charm/./src/charm.py:689: DeprecationWarning: Calling `ops.main.main()` is deprecated, call `ops.main()` instead
unit-grafana-agent-1: 20:03:57 WARNING unit.grafana-agent/1.tracing-relation-joined   main(GrafanaAgentMachineCharm)
unit-grafana-agent-1: 20:03:57 DEBUG unit.grafana-agent/1.juju-log tracing:46: ops 2.17.0 up and running.
unit-grafana-agent-1: 20:03:57 DEBUG unit.grafana-agent/1.juju-log tracing:46: failed to validate databag: {}
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/tempo_k8s/v2/tracing.py", line 209, in load
    return cls.parse_raw(json.dumps(data))  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 341, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 1 validation error for TracingProviderAppData
receivers
  field required (type=value_error.missing)
unit-grafana-agent-1: 20:03:57 INFO unit.grafana-agent/1.juju-log tracing:46: failed validating relation data for <ops.model.Relation tracing:46>