Size limit hit when saving the charm state to controller storage

[nobuto-m]

Bug Description

Not sure what's going on but the charm gives an error when the following steps are taken.

To Reproduce

1. deploy

   $ juju deploy cos-configuration-k8s cos-configuration
   Deployed "cos-configuration" from charm-hub charm "cos-configuration-k8s", revision 45 in channel latest/stable on ubuntu@20.04/stable

2. configure

   $ juju config cos-configuration \
   git_repo=https://github.com/nobuto-m/ceph.git \
   git_branch=cos-configuration-testing-main \
   grafana_dashboards_path=monitoring/ceph-mixin/dashboards_out/

Environment

cos-configuration-k8s latest/stable 45

Relevant log output

unit-cos-configuration-0: 11:48:14 INFO unit.cos-configuration/0.juju-log git-sync: I0404 11:48:14.230593      86 main.go:473] "level"=0 "msg"="starting up" "pid"=86 "args"=["/git-sync","--repo","https://github.com/nobuto-m/ceph.git","--branch","cos-configuration-testing-main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--one-time"]
unit-cos-configuration-0: 11:48:14 INFO unit.cos-configuration/0.juju-log Updating stored hash: git-sync hash changed from None (<class 'NoneType'>) to 0bfc7851eb297277581ed0a64d5c7a9b4824cc5f (<class 'str'>)
unit-cos-configuration-0: 11:48:14 DEBUG unit.cos-configuration/0.juju-log Alert rules path does not exist: /var/lib/juju/storage/content-from-git/0/repo/prometheus_alert_rules
unit-cos-configuration-0: 11:48:14 INFO unit.cos-configuration/0.juju-log Updating relation data with rule files from disk
unit-cos-configuration-0: 11:48:14 DEBUG unit.cos-configuration/0.juju-log storing reinit_without_topology_dropdowns: changed from [None] to [Done]
unit-cos-configuration-0: 11:48:14 DEBUG unit.cos-configuration/0.juju-log storing hash: changed from [None] to [0bfc7851eb297277581ed0a64d5c7a9b4824cc5f]
unit-cos-configuration-0: 11:48:15 DEBUG unit.cos-configuration/0.juju-log Alert rules path does not exist: /var/lib/juju/storage/content-from-git/0/repo/prometheus_alert_rules
unit-cos-configuration-0: 11:48:15 INFO unit.cos-configuration/0.juju-log Updating relation data with rule files from disk
unit-cos-configuration-0: 11:48:15 WARNING unit.cos-configuration/0.leader-elected ERROR max allowed value length (65536) exceeded
unit-cos-configuration-0: 11:48:15 ERROR unit.cos-configuration/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 473, in <module>
    main(COSConfigCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/main.py", line 443, in main
    framework.commit()
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 674, in commit
    self.on.commit.emit()
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 833, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 922, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 1027, in on_commit
    self.framework.save_snapshot(self)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/framework.py", line 710, in save_snapshot
    self._storage.save_snapshot(value.handle.path, data)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/storage.py", line 226, in save_snapshot
    self._backend.set(handle_path, snapshot_data)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/storage.py", line 365, in set
    _run(["state-set", "--file", "-"], input=content, check=True)
  File "/var/lib/juju/agents/unit-cos-configuration-0/charm/venv/ops/storage.py", line 48, in _run
    return subprocess.run([cmd, *args[1:]], encoding='utf-8', **kw)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/lib/juju/tools/unit-cos-configuration-0/state-set', '--file', '-']' returned non-zero exit status 1.
unit-cos-configuration-0: 11:48:15 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

cos-configuration-0.log

[nobuto-m]

This happens when trying to import: https://github.com/ceph/ceph/blob/19cc55c6259b87962367bcaf88d68c71334322a6/monitoring/ceph-mixin/dashboards_out/rgw-s3-analytics.json

By not having the JSON file in git_branch, then the charm does not error out.

[lucabello]

We should:

• try not using controller storage and see if that fixes it (it should, because the size is hardcoded; use a peer relation instead if necessary;
• else, we should figure out if the issue is related to the size limit in relation databags;
• if that's the case, figure out what do with the relation databag size limit.

(p.s., might be related to https://github.com/canonical/operator/pull/805)