Uncaught `ChangeError` exception when forming a log-proxy relation

[sed-i]

Bug Description

When I relate flog to Loki (via LogProxy),

1. an exception is printed to the status log
2. juju status shows active/idle for all
3. logs from flog do not reach loki

but the pebble plan is there

juju exec --unit flog/0 -- \
  PEBBLE_SOCKET=/charm/containers/workload/pebble.socket \
  pebble plan
services:
    flog:
        summary: flog service
        startup: enabled
        override: replace
        command: /bin/flog --format rfc5424 --loop --delay 1s --type log --output /bin/fake.log
    promtail:
        summary: promtail
        startup: disabled
        override: replace
        command: /opt/promtail/promtail-linux-amd64 -config.file=/etc/promtail/promtail_config.yaml

and promtail is running:

$ pgrep -la promtail        
1810916 /opt/promtail/promtail-linux-amd64 -config.file=/etc/promtail/promtail_config.yaml
1812372 /opt/promtail/promtail-linux-amd64 -config.file=/etc/promtail/promtail_config.yaml

However, nothing reaches loki:

$ juju show-unit loki/0 --format=json | jq -r '."loki/0".address' 
10.1.73.175

$ curl -G -s 10.1.73.175:3100/loki/api/v1/labels
{"status":"success","data":["__name__"]}

To Reproduce

1. deploy flog: juju deploy ./flog-k8s_ubuntu-20.04-amd64.charm flog --resource workload-image=mingrammer/flog
2. deploy loki: juju deploy --channel=edge loki-k8s loki
3. relate the two: juju relate loki flog

Environment

• loki-k8s edge 15
• juju 2.9.27

Relevant log output

unit-flog-0: 02:47:27.579 INFO unit.flog/0.juju-log log-proxy:4: Promtail binary zip file has been downloaded and stored in: /tmp/promtail-linux-amd64.zip
unit-flog-0: 02:47:28.102 DEBUG unit.flog/0.juju-log log-proxy:4: Promtail binary file has been downloaded.
unit-flog-0: 02:47:28.648 DEBUG unit.flog/0.juju-log log-proxy:4: The promtail binary file has been pushed to the workload container.
unit-flog-0: 02:47:28.704 ERROR unit.flog/0.juju-log log-proxy:4: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 84, in <module>
    main(FlogCharm)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-flog-0/charm/lib/charms/loki_k8s/v0/loki_push_api.py", line 1635, in _on_relation_changed
    self._setup_promtail()
  File "/var/lib/juju/agents/unit-flog-0/charm/lib/charms/loki_k8s/v0/loki_push_api.py", line 1984, in _setup_promtail
    self._container.restart(WORKLOAD_SERVICE_NAME)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/model.py", line 1152, in restart
    self._pebble.restart_services(service_names)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/pebble.py", line 1297, in restart_services
    return self._services_action('restart', services, timeout, delay)
  File "/var/lib/juju/agents/unit-flog-0/charm/venv/ops/pebble.py", line 1315, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "promtail" (cannot start service: fork/exec /opt/promtail/promtail-linux-amd64: no such file or directory)
----- Logs from task 0 -----
2022-03-25T02:47:28Z INFO Service "promtail" has never been started.
----- Logs from task 1 -----
2022-03-25T02:47:28Z ERROR cannot start service: fork/exec /opt/promtail/promtail-linux-amd64: no such file or directory
-----
unit-flog-0: 02:47:28.935 ERROR juju.worker.uniter.operation hook "log-proxy-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response

[mmanciop]

Browsing Docker Hub, it seems to me the flog container contains just a static go file (flog itself). Since that container does not have a shell, that may be the reason why we do not manage to tun promtail inside of it via LogProxy. Also, since the promtail we use is (very unfortunately, see #107) not statically compiled, it needs a compatible libc shared library to be in the container, and flog’s container does not have that.