On ARM, the service "agent" cannot start: fork/exec /bin/agent: exec format error.

[taurus-forever]

Bug Description

We are testing Charmed PostgreSQL K8s on ARM and grafana-agent-k8s throw an exception after relating with PostgreSQL:

unit-grafana-agent-k8s-0: 10:23:46 DEBUG unit.grafana-agent-k8s/0.juju-log Emitting Juju event agent_pebble_ready.
unit-grafana-agent-k8s-0: 10:23:46 ERROR unit.grafana-agent-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/./src/charm.py", line 242, in <module>
    main(GrafanaAgentK8sCharm)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/main.py", line 436, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/framework.py", line 942, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/./src/charm.py", line 120, in _on_agent_pebble_ready
    self._container.autostart()
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/model.py", line 2100, in autostart
    self._pebble.autostart_services()
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/pebble.py", line 1764, in autostart_services
    return self._services_action('autostart', [], timeout, delay)
  File "/var/lib/juju/agents/unit-grafana-agent-k8s-0/charm/venv/ops/pebble.py", line 1866, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "agent" (cannot start service: fork/exec /bin/agent: exec format error)
----- Logs from task 0 -----
2024-01-31T09:23:46Z ERROR cannot start service: fork/exec /bin/agent: exec format error

To Reproduce

On ARM !!!

juju deploy postgresql-k8s postgresql2-k8s --trust --channel 14/edge --config profile=testing --constraints arch=arm64

git clone https://github.com/canonical/grafana-agent-k8s-operator/
charmcraft pack
juju deploy ./grafana-agent-k8s_ubuntu-22.04-arm64.charm --constraints arch=arm64 --series jammy  --resource agent-image=ubuntu/grafana-agent:0.35.2-22.04_stable

juju integrate grafana-agent-k8s postgresql-k8s:grafana-dashboard
juju integrate grafana-agent-k8s postgresql-k8s:logging
juju integrate grafana-agent-k8s postgresql-k8s:metrics-endpoint

Environment

PINE64 Arch linux

[taurus@armjuju grafana-agent-k8s-operator]$ lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A55
    Model:               0
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r2p0
    CPU(s) scaling MHz:  100%
    CPU max MHz:         1800.0000
    CPU min MHz:         408.0000
    BogoMIPS:            48.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

Juju:

>>> snap list
Name                Version       Rev    Tracking       Publisher        Notes
charmcraft          2.5.4         2097   latest/stable  canonical**      classic
charmed-postgresql  14.10         97     14/edge/…      dataplatformbot  -
core                16-2.60.4     16204  latest/stable  canonical**      core
core20              20231123      2107   latest/stable  canonical**      base
core22              20231123      1035   latest/stable  canonical**      base
jhack               0.3.28        283    latest/edge    ppasotti         -
juju                3.1.7         25752  3.1/stable     canonical**      -
lxd                 5.19-8635f82  26202  latest/stable  canonical**      -

Microk8s doesn't work on this kernel (no apparmor) using K3s, but "should work anyway (c)".

Relevant log output

See above in the bugreport.

Additional context

ARM support for grafana-agent-k8s introduced here: https://github.com/canonical/grafana-agent-k8s-operator/pull/264

[sed-i]

Arm is a future roadmap item. Closing for now.