promtail binary does not have journald slurping enabled

Bug Description

Grafana Agent is not slurping logs from journald as expected; the Grafana Agent issues a warning that a journal target is configured, but the promtail binary has not been compiled with this option enabled. All configuration was performed according to https://charmhub.io/topics/canonical-observability-stack/tutorials/instrumenting-machine-charms.

I am running a modified version of landscape-scalable with the CosAgentProvider included. I've copied the relevant modifications below.

# metadata.yml
provides:
  ...
  cos-agent:
    interface: cos_agent

# charm.py
from charms.grafana_agent.v0.cos_agent import COSAgentProvider
...
class LandscapeServerCharm(CharmBase):
    """Charm the service."""

    def __init__(self, *args):
        super().__init__(*args)
        ...
        self._grafana_agent = COSAgentProvider(self)

To Reproduce

juju deploy ./landscape-server_ubuntu-22.04-amd64-arm64_ubuntu-20.04-amd64-arm64.charm
juju deploy grafana-agent --channel edge
juju relate grafana-agent landscape-server

Assuming a microk8s cloud running COS called microk8s-cos

juju consume microk8s-cos:admin/cos.prometheus-receive-remote-write
juju consume microk8s-cos:admin/cos.loki-logging
juju consume microk8s-cos:admin/cos.grafana-dashboards

juju relate grafana-agent prometheus-receive-remote-write
juju relate grafana-agent loki-logging
juju relate grafana-agent grafana-dashboards

Verify that error message appears by restarting the application container and checking journald:

lxc restart <container>
juju ssh <application unit>
journalctl -b -x -u snap.grafana-agent.grafana-agent | grep journal

Environment

Running juju version=3.1.6-genericlinux-amd64 locally

Relevant log output

<application container>:~$ journalctl -b -x -u snap.grafana-agent.grafana-agent | grep journal
Oct 25 14:39:45 juju-561bfc-15 grafana-agent.grafana-agent[272]: ts=2023-10-25T14:39:45.567668197Z caller=journaltargetmanager.go:29 level=warn component=logs logs_config=log_file_scraper msg="WARNING!!! Journal target was configured but support for reading the systemd journal is not compiled into this build of promtail!"

Additional context

No response

canonical / grafana-agent-operator