search

canonical / operator

Pure Python framework for writing Juju charms
Apache License 2.0
245 stars 119 forks source link

Document valid schema for pod's set_spec function and how to set environment variables #463

Closed alexellis closed 3 years ago

alexellis commented 3 years ago

I am trying without any success to add an environment variable to a charm I'm building for OpenFaaS:

"env" (as per the Kubernetes schema) fails

"envConfig" as seen in the Istio example also fails.

https://github.com/juju-solutions/bundle-istio/blob/master/charms/istio-ingressgateway/reactive/charm.py#L130

ops.model.ModelError: b'ERROR json: unknown field "env"\n'
ops.model.ModelError: b'ERROR json: unknown field "envConfig"\n'
ops.model.ModelError: b'ERROR json: unknown field "envs"\n'

Attempted Charm:

    def _on_config_changed(self, _=None):
        pod_spec = self._build_pod_spec()
        self.model.pod.set_spec(pod_spec)
        self.unit.status = ActiveStatus("OpenFaaS pod ready.")

    def _build_pod_spec(self):
        spec = {
            "containers": [
                {
                    "name": self.app.name,
                    "imageDetails": {"imagePath": "openfaas/gateway:0.20.2"},
                    "ports": [{"containerPort": 8080, "protocol": "TCP"}],
                    "files": [],
                    "config": {},  # used to store hashes of config file text
                    "env": [
                        {"functions_provider_url": "192.168.0.15:8081"},
                        {"direct_functions": "false"},
                        {"basic_auth": "false"},
                        {"faas_prometheus_host": "192.168.0.35"},
                        {"faas_prometheus_port": "9090"},
                    ]
                }
            ]
        }

Kubernetes schema:

      containers:
      - name: gateway
        resources:
            requests:
              cpu: 50m
              memory: 120Mi
        image: openfaas/gateway:0.20.2
        imagePullPolicy: Always
        securityContext:
          readOnlyRootFilesystem: true
          runAsUser: 10001
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          timeoutSeconds: 5
        env:
        - name: read_timeout
          value: "65s"
        - name: write_timeout
          value: "65s"
        - name: upstream_timeout
camille-rodriguez commented 3 years ago

Hi @alexellis , Idk why, but the env variables are named "envConfig" in the pod spec. This example (metallb-speaker operator) uses it : https://github.com/charmed-kubernetes/metallb-operator/blob/master/charms/metallb-speaker/src/charm.py#L127.

'containers': [{
    'name': 'speaker',
    'imageDetails': image_info,
    'imagePullPolicy': 'Always',
    'ports': [{
        'containerPort': 7472,
        'protocol': 'TCP',
        'name': 'monitoring'
    }],
    'envConfig': {
        'METALLB_NODE_NAME': {
            'field': {
                'path': 'spec.nodeName',
                'api-version': 'v1'
            }
        },
}]

I have also found useful information in discourse https://discourse.charmhub.io/t/k8s-spec-v3-changes/2698.

Hope it helps! Cheers

PS. I see that you tried envConfig... it might be a formatting error then, I would double check where you've used [] or {} compared to an example

alexellis commented 3 years ago

Unfortunately it didn't seem to work when I tried it above? Does my example look off? I tried all three: env, envs and envConfig.

If the Operator framework uses a non-standard spec, where can we find the schema documented?

alexellis commented 3 years ago

Did you find a good workflow for the dev/test flow of initial deploy/edit/observe/edit/re-deploy? I was finding that juju upgrade-charm openfaas --path=./openfaas.charm was not working if the deployed version had a syntax error of exception, instead it seems the workaround is to run juju add-model NAME every time to be able to deploy a new version.

Error when switching over to a map:

        pod_spec = self._build_pod_spec()
        self.model.pod.set_spec(pod_spec)
        self.unit.status = ActiveStatus("OpenFaaS pod ready.")

    def _build_pod_spec(self):
        spec = {
            "containers": [
                {
                    "name": self.app.name,
                    "imageDetails": {"imagePath": "openfaas/gateway:0.20.2"},
                    "ports": [{"containerPort": 8080, "protocol": "TCP"}],
                    "files": [],
                    "config": {},  # used to store hashes of config file text
                    "envConfig": {
                        "functions_provider_url": "192.168.0.15:8081",
                        "direct_functions": "false",
                        "basic_auth": "false",
                        "faas_prometheus_host": "192.168.0.35",
                        "faas_prometheus_port": "9090",
                    }
                }
            ]
        }

        return spec

Gives:

2020-12-07 21:00:17 ERROR juju.worker.uniter.operation runhook.go:136 hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
2020-12-07 21:00:17 INFO juju.worker.uniter resolver.go:143 awaiting error resolution for "config-changed" hook
2020-12-07 21:00:22 INFO juju.worker.uniter resolver.go:143 awaiting error resolution for "config-changed" hook
2020-12-07 21:00:23 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/model.py", line 1049, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-openfaas-0/pod-spec-set', '--file', '/tmp/tmp704lsj4z-pod-spec-set/spec.yaml')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 50, in <module>
    main(OpenfaasCharm)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/main.py", line 402, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 23, in _on_config_changed
    self.model.pod.set_spec(pod_spec)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/model.py", line 926, in set_spec
    self._backend.pod_spec_set(spec, k8s_resources)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/model.py", line 1154, in pod_spec_set
    self._run('pod-spec-set', *args)
  File "/var/lib/juju/agents/unit-openfaas-0/charm/venv/ops/model.py", line 1051, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR json: unknown field "envConfig"\n'
2020-12-07 21:00:23 ERROR juju.worker.uniter.operation runhook.go:136 hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
2020-12-07 21:00:23 INFO juju.worker.uniter resolver.go:143 awaiting error resolution for "config-changed" hook
^C
alexellis commented 3 years ago

Unfortunately that code gives the error I shared above, would you be open to giving it a try to see if you can figure out what I've missed? https://github.com/alexellis/openfaas-charm-example

camille-rodriguez commented 3 years ago

So in your code you're not specifying the version of pod_spec you want to use, which leads to using (I believe) version 2, where envConfig did not exist yet. Once changing to version 3, you need to remove "config", which doesn't exist, and "files" as well. So with this :

    def _build_pod_spec(self):
        spec = {
            "version": 3,
            "containers": [
                {
                    "name": self.app.name,
                    "imageDetails": {"imagePath": "openfaas/gateway:0.20.2"},
                    "ports": [{"containerPort": 8080, "protocol": "TCP"}],
                    #"files": [],
                    "envConfig": {
                        "functions_provider_url": "192.168.0.15:8081",
                        "direct_functions": "false",
                        "basic_auth": "false",
                        "faas_prometheus_host": "192.168.0.35",
                        "faas_prometheus_port": "9090"
                    }
                }
            ]
        }

I got a green status 

$ juju status
Model  Controller  Cloud/Region        Version  SLA          Timestamp
t3     microk8s    microk8s/localhost  2.8.6    unsupported  16:34:55-05:00

App       Version         Status   Scale  Charm     Store  Rev  OS          Address         Notes
openfaas  gateway:0.20.2  waiting      1  openfaas  local    0  kubernetes  10.152.183.130  

Unit         Workload  Agent  Address      Ports     Message
openfaas/0*  active    idle   10.1.142.16  8080/TCP  OpenFaaS pod ready.

:)

camille-rodriguez commented 3 years ago

Did you find a good workflow for the dev/test flow of initial deploy/edit/observe/edit/re-deploy? I was finding that juju upgrade-charm openfaas --path=./openfaas.charm was not working if the deployed version had a syntax error of exception, instead it seems the workaround is to run juju add-model NAME every time to be able to deploy a new version.

Regarding that, I typically redeploy in a new model after making changes to my code. There might be a faster way to test when code breaks, you could ask in the irc channel #smooth-operator, where the framework developers hang out.

camille-rodriguez commented 3 years ago

If that solution worked for you as well I would suggest to close this bug.

alexellis commented 3 years ago

"version": 3, was the missing piece, however I feel like we need a new issue like : "Define a Pod Spec for V1, V2 and V3"

Thanks @camille-rodriguez for the assist.

lukemarsden commented 3 years ago

Read together, these documents go some way to documenting a spec:

e.g. to find the command value, I looked in the v1 spec, and assumed that the v3 builds on v2 which builds on v1.

lukemarsden commented 3 years ago

There is also a spec here! https://juju.is/docs/charm-writing/kubernetes