Pebble fails to add a layer for Loki log forwarding.

[Abuelodelanada]

I'm having an issue with an integration test. With juju 3.3.3 the test fails. With juju 3.4.0 the same integration test pass.

But as far as I understand the issue is not with juju itself but with pebble which fails to add a layer (A Loki log forwarding layer in fact).

I've checked in a local deployments and found that charms deployed using Juju 3.3.3 have pebblev1.4.1, and charms deployed using juju 3.4.0 have v1.7.0

unit-log-forwarder-tester-0: 15:17:16.817 INFO unit.log-forwarder-tester/0.juju-log log-forwarder:1: The following layer will be added:
unit-log-forwarder-tester-0: 15:17:16.820 INFO unit.log-forwarder-tester/0.juju-log log-forwarder:1: log-targets:
  loki/0:
    labels:
      charm: log-forwarder-tester
      juju_application: log-forwarder-tester
      juju_model: test-log-forwarder-ee97
      juju_model_uuid: f525925c-59fd-49e3-87d3-48780aae47fd
      juju_unit: log-forwarder-tester/0
      product: Juju
    location: http://loki-0.loki-endpoints.test-log-forwarder-ee97.svc.cluster.local:3100/loki/api/v1/push
    override: replace
    services:
    - all
    type: loki

unit-log-forwarder-tester-0: 15:17:16.850 ERROR unit.log-forwarder-tester/0.juju-log log-forwarder:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 100, in <module>
    main(LogForwarderTesterCharm)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/main.py", line 456, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/framework.py", line 865, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/framework.py", line 955, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/lib/charms/loki_k8s/v1/loki_push_api.py", line 2539, in _update_logging
    self._update_endpoints(container, loki_endpoints)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/lib/charms/loki_k8s/v1/loki_push_api.py", line 2556, in _update_endpoints
    _PebbleLogClient.enable_endpoints(
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/lib/charms/loki_k8s/v1/loki_push_api.py", line 2490, in enable_endpoints
    container.add_layer(f"{container.name}-log-forwarding", layer, combine=True)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/model.py", line 2169, in add_layer
    self._pebble.add_layer(label, layer, combine=combine)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/pebble.py", line 2101, in add_layer
    self._request('POST', '/v1/layers', body=body)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/pebble.py", line 1772, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-log-forwarder-tester-0/charm/venv/ops/pebble.py", line 1818, in _request_raw
    raise APIError(body, code, status, message) from None
ops.pebble.APIError: cannot parse layer YAML: cannot parse layer "workload-a-log-forwarding": yaml: unmarshal errors:
  line 3: field labels not found in type plan.LogTarget
unit-log-forwarder-tester-0: 15:17:17.066 ERROR juju.worker.uniter.operation hook "log-forwarder-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1

How to reproduce it:

1. Clone Loki repo: git@github.com:canonical/loki-k8s-operator.git
2. switch to pebble_ready branch: git switch pebble_ready
3. Run the itest with: tox -e integration -- -k test_log_forwarder.py

Repeat these steps in both juju 3.3.3 and juju 3.4.0 deployments

[benhoyt]

As you point out, Juju 3.3.3 uses Pebble v1.4.1, which doesn't have the log labels feature yet -- log labels were only introduced in Pebble v1.6.0. Juju 3.4.0 uses Pebble v1.7.0, so it has support for log labels. So unfortunately to fix this, you'll need to upgrade to Juju 3.4.x, or avoid using log labels.

[Abuelodelanada]

As you point out, Juju 3.3.3 uses Pebble v1.4.1, which doesn't have the log labels feature yet -- log labels were only introduced in Pebble v1.6.0. Juju 3.4.0 uses Pebble v1.7.0, so it has support for log labels. So unfortunately to fix this, you'll need to upgrade to Juju 3.4.x, or avoid using log labels.

Thanks @benhoyt!

I thought labels were in v1.4.1 too!

[benhoyt]

Log forwarding itself (to Loki) was introduced in v1.4.0, but not log labels, sorry!