Automatically transform config into pod spec

[knkski]

In working on the charms in the Kubeflow bundle, one pattern that has emerged for all of them is that they're actually just some Python wrappers around watching for any changes to state and rerendering some YAML that gets handed off to pod_spec_set, e.g. the ambassador charm:

https://github.com/juju-solutions/bundle-kubeflow/blob/master/charms/ambassador/reactive/ambassador.py

Would it be possible to support just writing a charm as a pod spec set template that gets rerendered whenever state changes (e.g. config changes, relation changes)? This would make handling state trivial in a charm, as it then manages none itself.

[jameinel]

I don't think we would ever have just a charm that is just yaml. It shouldn't be very hard to set up a bit of python code that grabs the various related data (config data, relation data) and then reads a known yaml file, and renders it. If we have a clear pattern of this, it might be something we would pull into the framework, but in the mid term any charm can just use a common python class to do that work.

However, rendering a YAML template doesn't seem particularly more interesting/useful than a python dict. Certainly there will always need to be some sort of mapping logic that figures out where the content is coming from (is it charm config, is it relation data, is it derived data, is it an environment variable).

I think there will always be a little bit of python that maps from sources of data to how the template is rendered. You can make that common shared code, but it still needs to exist.

On Thu, Jan 23, 2020 at 9:55 PM Kenneth Koski notifications@github.com wrote:

In working on the charms in the Kubeflow bundle, one pattern that has emerged for all of them is that they're actually just some Python wrappers around watching for any changes to state and rerendering some YAML that gets handed off to pod_spec_set, e.g. the ambassador charm:

https://github.com/juju-solutions/bundle-kubeflow/blob/master/charms/ambassador/reactive/ambassador.py

Would it be possible to support just writing a charm as a pod spec set template that gets rerendered whenever state changes (e.g. config changes, relation changes)? This would make handling state trivial in a charm, as it then manages none itself.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/canonical/operator/issues/113?email_source=notifications&email_token=AABRQ7IISVHJAZRTX2T6T5LQ7HY2DA5CNFSM4KK4C4NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IILBITQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABRQ7I55SIAL3253LZI5NTQ7HY2DANCNFSM4KK4C4NA .

[knkski]

I don't think individual charms need the mapping logic. For example, you could have a template that looks something like this:

containers:
- name: foo-container
  environment:
    BAR_HOST: {{ state.relations.bar.juju.host }}
    BAR_PORT: {{ state.relations.bar.config.port }}
  ports:
    - name: http
      containerPort: {{ state.config.port }}

In other words, the template itself has access to all the various forms of state that it needs. Whether that state gets presented to the template via Juju itself or a charming library is an implementation detail to charm authors, though. They just don't have to implement that logic themselves. The declarative nature of YAML means writing simple charms would be trivial and less boilerplate vs. writing them in Python.

Which isn't to say that one would be forced to use YAML. It would still make sense to support handling more complex use cases with Python, but given how many of the Kubeflow charms have turned out to just be simple wrappers around rerendering a template whenever state changes, it seems like charms as YAML would capture the majority of charming use cases.

[johnsca]

I tend to agree with John that even for the most trivial charms reading a small bit of Python vs YAML templated in an arbitrary templating language is mostly a matter of taste and unlikely to really be more readable one way or the other, and that even a little bit of complexity will almost certainly lead to wanting to move logic into the Python anyway.

But there's also nothing stopping you from right now creating a reusable component, perhaps one that implements a CharmBase subclass that you can use directly as your entry point, which defines the semantics that feel comfortable to you and moving forward with it. If it works for others as well, we can work it into whatever inclusion / discoverability solution we end up with. (There's some discussion that needs to be had on that, particularly after an off-hand comment by Mark during the sprint that bears some serious consideration but likely requires working through some tricky details.)

[timClicks]

Perhaps another way to frame this would be whether we could create charms with a code generator? We have charm templates from charm-tools. Perhaps charm create could be parameterised?

[jameinel]

I'm less concerned about a charm create than having a nice base class for it, as the class is necessary even if you create, and likely to be 90% of the solution.

On Tue, Feb 4, 2020 at 3:28 AM Tim McNamara notifications@github.com wrote:

Perhaps another way to frame this would be whether we could create charms with a code generator? We have charm templates from charm-tools https://github.com/juju/charm-tools. Perhaps charm create could be parameterised?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/canonical/operator/issues/113?email_source=notifications&email_token=AABRQ7O7JHVF3FVL6OSYPS3RBCSBTA5CNFSM4KK4C4NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVZEUQ#issuecomment-581669458, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABRQ7PDKQNOTG4V2IYSCW3RBCSBTANCNFSM4KK4C4NA .

[zicklag]

I'm not saying to go one way or the other with this but an interesting project you could look at is ytt which allows you to create a YAML template with an embedded, extremely Python-ish, scripting language called starlark

[niemeyer]

There are at least three different aspects that we can look into individually:

1. A well integrated templating module might be adopted (filed as issue #228)
2. The pod spec API should be improved at some point (filed as issue #229)
3. There might be an automated way for juju to transform configuration into a pod spec (not filed)

In my view, points 1 and 2 sound like good improvements and they both make the life of creating a pod spec more comfortable. Point 3 doesn't sound so good, though, because we don't want to encourage everyone to just have the k8s spec stamped as the charm configuration. Instead, the charm will often want to do some sort of manipulation, including taking relations (potentially multiple), configuration, and other knowledge about the problem space into account before creating the pod spec, so the framework should make it very comfortable to do that, but shouldn't be encouraging a passthrough behavior.

Since that latter point seems to be one of the key points in this issue, I'm closing it for now, but the other related and more specific issues will remain open until we address these other aspects.