Can't send environment to pebble

[mthaddon]

I'm working on converting a charm to use pebble (as a proof of concept) and am running into problems trying to send it an environment configuration.

The pebble docs specify:

        environment:
            - VAR1: val1
            - VAR2: val2
            - VAR3: val3

So I tried passing a list of dictionaries to the environment config, but I got:

unit-gunicorn-0: 10:04:56 INFO unit.gunicorn/0.juju-log About to dump yaml config <<EOM
description: gunicorn layer
services:
  gunicorn:
    command: /srv/gunicorn/run
    default: start
    environment:
    - FAVOURITEFOOD: burgers
    - FAVOURITEDRINK: ale
    override: replace
    summary: gunicorn service
summary: gunicorn layer

EOM
unit-gunicorn-0: 10:04:56 ERROR unit.gunicorn/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 438, in <module>
    main(GunicornK8sCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 406, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 137, in _on_gunicorn_workload_ready
    container.add_layer("gunicorn", pebble_config)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/model.py", line 1065, in add_layer
    self._pebble.add_layer(label, layer, combine=combine)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 668, in add_layer
    layer_yaml = Layer(layer).to_yaml()
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 427, in __init__
    self.services = {name: Service(name, service)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 427, in <dictcomp>
    self.services = {name: Service(name, service)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 463, in __init__
    self.environment = dict(raw.get('environment') or {})
ValueError: dictionary update sequence element #0 has length 1; 2 is required

Looking at the code for the operator framework self.environment = dict(raw.get('environment') or {}) so I tried just passing in a dictionary, but I got the following:

unit-gunicorn-0: 10:18:33 ERROR juju.worker.uniter pebble poll failed: hook failed
unit-gunicorn-0: 10:18:37 INFO unit.gunicorn/0.juju-log About to dump yaml config <<EOM
description: gunicorn layer
services:
  gunicorn:
    command: /srv/gunicorn/run
    default: start
    environment:
      FAVOURITEDRINK: ale
      FAVOURITEFOOD: burgers
    override: replace
    summary: gunicorn service
summary: gunicorn layer

EOM
unit-gunicorn-0: 10:18:37 ERROR unit.gunicorn/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 531, in _request
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 450, in <module>
    main(GunicornK8sCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 406, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 150, in _on_gunicorn_workload_ready
    container.add_layer("gunicorn", pebble_config)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/model.py", line 1065, in add_layer
    self._pebble.add_layer(label, layer, combine=combine)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 682, in add_layer
    self._request('POST', '/v1/layers', body=body)
  File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 542, in _request
    raise APIError(body, code, status, message)
ops.pebble.APIError: cannot parse layer YAML: cannot parse layer "gunicorn": yaml: unmarshal errors:
  line 7: cannot unmarshal !!map into []plan.StringVariable
unit-gunicorn-0: 10:18:38 ERROR juju.worker.uniter.operation hook "gunicorn-workload-ready" (via hook dispatching script: dispatch) failed: exit status 1

[jameinel]

Sounds like Pebble expects a list of key: value but the python code is expecting a dictionary. Offhand I dont know why you would want a list, as env vars can't be repeated and their order shouldn't matter, but maybe there is a reason I'm missing.

John =:->

On Mon, Mar 15, 2021, 05:25 mthaddon @.***> wrote:

I'm working on converting a charm to use pebble (as a proof of concept) and am running into problems trying to send it an environment configuration.

The pebble docs specify:

    environment:
        - VAR1: val1
        - VAR2: val2
        - VAR3: val3

So I tried passing a list of dictionaries to the environment config, but I got:

unit-gunicorn-0: 10:04:56 INFO unit.gunicorn/0.juju-log About to dump yaml config <<EOM description: gunicorn layer services: gunicorn: command: /srv/gunicorn/run default: start environment:

• FAVOURITEFOOD: burgers
• FAVOURITEDRINK: ale override: replace summary: gunicorn service summary: gunicorn layer

EOM unit-gunicorn-0: 10:04:56 ERROR unit.gunicorn/0.juju-log Uncaught exception while in charm code: Traceback (most recent call last): File "./src/charm.py", line 438, in main(GunicornK8sCharm, use_juju_for_storage=True) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 406, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 140, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 278, in emit framework._emit(event) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 722, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 767, in _reemit custom_handler(event) File "./src/charm.py", line 137, in _on_gunicorn_workload_ready container.add_layer("gunicorn", pebble_config) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/model.py", line 1065, in add_layer self._pebble.add_layer(label, layer, combine=combine) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 668, in add_layer layer_yaml = Layer(layer).to_yaml() File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 427, in init self.services = {name: Service(name, service) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 427, in self.services = {name: Service(name, service) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 463, in init self.environment = dict(raw.get('environment') or {}) ValueError: dictionary update sequence element #0 has length 1; 2 is required

Looking at the code for the operator framework self.environment = dict(raw.get('environment') or {}) so I tried just passing in a dictionary, but I got the following:

unit-gunicorn-0: 10:18:33 ERROR juju.worker.uniter pebble poll failed: hook failed unit-gunicorn-0: 10:18:37 INFO unit.gunicorn/0.juju-log About to dump yaml config <<EOM description: gunicorn layer services: gunicorn: command: /srv/gunicorn/run default: start environment: FAVOURITEDRINK: ale FAVOURITEFOOD: burgers override: replace summary: gunicorn service summary: gunicorn layer

EOM unit-gunicorn-0: 10:18:37 ERROR unit.gunicorn/0.juju-log Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 531, in _request response = self.opener.open(request, timeout=self.timeout) File "/usr/lib/python3.8/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response response = self.parent.error( File "/usr/lib/python3.8/urllib/request.py", line 569, in error return self._call_chain(args) File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(args) File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 400: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./src/charm.py", line 450, in main(GunicornK8sCharm, use_juju_for_storage=True) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 406, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/main.py", line 140, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 278, in emit framework._emit(event) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 722, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/framework.py", line 767, in _reemit custom_handler(event) File "./src/charm.py", line 150, in _on_gunicorn_workload_ready container.add_layer("gunicorn", pebble_config) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/model.py", line 1065, in add_layer self._pebble.add_layer(label, layer, combine=combine) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 682, in add_layer self._request('POST', '/v1/layers', body=body) File "/var/lib/juju/agents/unit-gunicorn-0/charm/venv/ops/pebble.py", line 542, in _request raise APIError(body, code, status, message) ops.pebble.APIError: cannot parse layer YAML: cannot parse layer "gunicorn": yaml: unmarshal errors: line 7: cannot unmarshal !!map into []plan.StringVariable unit-gunicorn-0: 10:18:38 ERROR juju.worker.uniter.operation hook "gunicorn-workload-ready" (via hook dispatching script: dispatch) failed: exit status 1

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/canonical/operator/issues/486, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABRQ7NVJBHPTEP2Z3IISGDTDXHA5ANCNFSM4ZGFIZIA .

[benhoyt]

Hmm, yeah, this is non-intuitive and a bit weird. What's documented at https://github.com/canonical/pebble#layer-configuration does actually work, but note that you need an extra - in front of the key/value pairs, like so:

    environment:
      - FAVOURITEFOOD: burgers
      - FAVOURITEDRINK: ale

However, then when you execute get_plan or pebble plan, it spits back out the following:

        environment:
            - name: FAVOURITEDRINK
              value: ale
            - name: FAVOURITEFOOD
              value: burgers

But that format isn't accepted as input. I'll dig in further in the next couple of days and we'll get this fixed in Pebble (and Python Operator Framework if need be).

CC: @niemeyer

[benhoyt]

For some reason it's done this way explicitly, see code. We have a custom unmarshaller, but I didn't add the corresponding marshaller when adding the get-plan command. IMO we should switch to a plain object for "environment". I've asked Gustavo for input.

[niemeyer]

The issue with plain objects is that environment variables may have order depending on how we allow them to expand. That's why it's a sequence. We can switch to a map and parse that in Go taking the order into account, but that's not a feature that follows the spec. We've done that before, so we can also choose to do it here again if there's consensus, but we need to at least verify that Python would be able to parse it correctly, for example.

Otherwise, you're on the right track with the marshaler.

[benhoyt]

This is fixed now in the latest Pebble and Python Operator Framework. Pebble accepts and returns an array of objects, like so:

    environment:
      - FAVOURITEFOOD: burgers
      - FAVOURITEDRINK: ale

And in Python-land in the Service.environment attribute, it's serialized to a list of 2-tuples, like so:

[('FAVOURITEFOOD', 'burgers'), ('FAVOURITEDRINK', 'ale')]

Like your food preferences, by the way. ;-)