Add `cross_model` property to `model.Relation`

[sed-i]

Our charm code needs to know which URL to advertise over relation data, so that remote app's workload would be able to reach our charm. For example, for in-model relation we would advertise the FQDN. For cross-model, we'd advertise the ingress URL.

Not sure if/where this is exposed by Juju:

https://github.com/juju/juju/blob/2b2308500d71d17867256cc81dce9dbce877bcc4/rpc/params/applications.go#L497

CrossModel       bool                    `json:"cross-model"`

[sed-i]

Just wondering what's the prospect for this feature @benhoyt. Seems like it is becoming critical to our TLS story: the same COS charm (e.g. prometheus) needs to advertise different URLs depending on where it is deployed relative to grafana:

graph LR

subgraph COS model
  prometheus ---|k8s fqdn| grafana

end
external-prometheus ---|ingress URL| grafana

1. COS charms generate CSRs with the k8s fqdn as the SAN DNS, and a local CA signs.
2. All COS charms trust the local CA.
3. COS charms also have an ingress URL provided by traefik.
4. Traefik's CSR is signed by an external-ca.

The problem is that we must now choose what URL COS charms advertise to each other (e.g. grafana datasource in prometheus):

• If we use fqdn, then TLS works but CMRs do not.
• If we use ingress, CMRs work, but now we'd need to have COS charms also trust the external CA.

Seems like it would make sense if charms could advertise fqdn when inside the same cluster, but ingress when cross cluster.

Other potential solutions:

• Have the (permissive) local ca (blindly) sign both fqdn and the ingress url. This is an antipattern.
• Somehow utilize SNI, as @tlm proposed. Not sure if this is feasible here.
• Propagate the external CA cert into all relevant COS charms, e.g. by directly relating them to the external CA (antipattern) or have traefik forward its ca cert to all proxied apps (possibly an antipattern?).
• Assume the external-ca is signed by a root CA, and that all charms trust all root CAs. This way no transfer of certs would be needed because the external CA would have a verifiable chain of trust. But this would make charms more difficult to test in CI because the stand-in external-ca would not have a chain of trust.

cc: @simskij @gruyaume @deusebio

[tlm]

Hi @sed-i,

I have had a read of the above. Thanks for the awesome write up of the problem. This is going to be an ongoing discussion but I do have some initial thoughts, statements and ideas.

• I hope you don't mind if we clarify some important terms when starting out. In the example above our CA's involved are only signing one of an FQDN or ip address. It is not possible to sign URL's as that is a concept much higher in the protocol then one at which TLS operates at.

• In the example above we have Grafana talking to both an internal (internal to the cluster) Prometheus and an external Prometheus. Logically I can understand the need to send Grafana to the Prometheus inside the same cluster using the in cluster FQDN but to keep it simple why not just send all communication via Traefik ? You're not creating a bottle neck and assuming Traefik is deployed in Kubernetes with the same stack then the ip address Traefik resolves to for whatever the FQDN is still within the same route table as the local cluster.

• My understanding of the problem here and please correct me if I am wrong. When performing relations with other applications you are faced with two choices. Do I tell this other end to come in via myapp.cluster.local or via myapp.example.com. My answer for this would be if it's not directly in the charm then always provide myapp.example.com.

Just to head off a few questions/problems I can see I would suggest the following:

• The certificate that a unit application is using by requesting one from from the local ca should be transferred to Traefik when relating to the ingress controller. That way you get end to end encryption through the proxy/load balancer.
• If Traefik has not been setup with an external FQDN we could put a cluster local one into it perhaps until one is established?
• Traefik is able to use local cluster CA's for requesting certificates. This can be used in place of an externally trusted CA till one can be trusted and would be the same one you are currently trusting at the moment. Traefik will pick the correct certificate to give the client based on the hostname they present in the TLS hello request.

Hope the above starts to help with the problem. Can I suggest that as we discuss this more we look to break down the problem into smaller chunks. Hopefully that will make it easier to write up solutions as well.

Cheers tlm

[sed-i]

Thanks @tlm!

I think that if we always go through traefik then we're at the same time creating a single point of failure (traefik) and polluting the relation space with N new relations (receive-ca-cert).

---

why not just send all communication via Traefik ?

We have end-to-end TLS in place, so when grafana queries prometheus and goes through traefik, then grafana must trust the CA that signed traefik. This means we need to implement a mechanism for forwarding the external ca cert to not only traefik itself, but also to all relevant charms under traefik (grafana, prometheus, alertmanager). This is certainly possible, but would:

• "Pollute" the relations space. Also, deploying would be more error prone (admin forgot one juju relate command).
• Seemingly violate separation of concerns: external ca is for traefik, internal ca is for same-model charms. (We would still need to forward the external ca cert to the out-of-model prom/other app though.)

You're not creating a bottle neck

Afaict, if all charms always go through the traefik ingress url, then we do create a bottleneck/single point of failure.

The certificate that a unit application is using by requesting one from from the local ca should be transferred to Traefik when relating to the ingress controller.

Yes, we already have that in place.

If Traefik has not been setup with an external FQDN we could put a cluster local one into it perhaps until one is established?

I'm not clear on the details of this, but currently for testing purposes we use a bare IP address in the SAN (having some issues with it currently actually).

Traefik is able to use local cluster CA's for requesting certificates. This can be used in place of an externally trusted CA

This is exactly what we've been doing until recently, but it resulted is some blindspots in the implementation. So now we're switching to local ca + external ca on purpose.

[benhoyt]

Update: @tlm was going to put together some notes / a spec on the networking aspects of this to see if we can converge on a proposal.

For reference, the Juju team feels strongly this shouldn't be exposed as a "cross model" property on the relation, as charms must remain agnostic to such deployment concerns and get the networking info they need to know from Juju. CC: @wallyworld

[simskij]

Update: @tlm was going to put together some notes / a spec on the networking aspects of this to see if we can converge on a proposal.

For reference, the Juju team feels strongly this shouldn't be exposed as a "cross model" property on the relation, as charms must remain agnostic to such deployment concerns and get the networking info they need to know from Juju. CC: @wallyworld

I strongly believe that the Juju team (@wallyworld, @manadart, @hpidcock et al) needs to provide us with a reasonable alternative in that case. That stance works well for machine deployments, but less well on Kubernetes where ingresses, cluster-internal dns and what not complicates things.

Until Juju grows the ability to provide ingress, or be aware of it, as well as determining whether ingress should or should not be used based on which charm is trying to connect to which, that stance is unfortunately more or less equal to "nah". 😅

[simskij]

Update: @tlm was going to put together some notes / a spec on the networking aspects of this to see if we can converge on a proposal. For reference, the Juju team feels strongly this shouldn't be exposed as a "cross model" property on the relation, as charms must remain agnostic to such deployment concerns and get the networking info they need to know from Juju. CC: @wallyworld

I strongly believe that the Juju team (@wallyworld, @manadart, @hpidcock et al) needs to provide us with a reasonable alternative in that case. That stance works well for machine deployments, but less well on Kubernetes where ingresses, cluster-internal dns and what not complicates things.

Until Juju grows the ability to provide ingress, or be aware of it, as well as determining whether ingress should or should not be used based on which charm is trying to connect to which, that stance is unfortunately more or less equal to "nah". 😅

Bumping this as this is still an actual problem for us with no good solution.

[wallyworld]

Update: @tlm was going to put together some notes / a spec on the networking aspects of this to see if we can converge on a proposal. For reference, the Juju team feels strongly this shouldn't be exposed as a "cross model" property on the relation, as charms must remain agnostic to such deployment concerns and get the networking info they need to know from Juju. CC: @wallyworld

I strongly believe that the Juju team (@wallyworld, @manadart, @hpidcock et al) needs to provide us with a reasonable alternative in that case. That stance works well for machine deployments, but less well on Kubernetes where ingresses, cluster-internal dns and what not complicates things. Until Juju grows the ability to provide ingress, or be aware of it, as well as determining whether ingress should or should not be used based on which charm is trying to connect to which, that stance is unfortunately more or less equal to "nah". 😅

Bumping this as this is still an actual problem for us with no good solution.

@simskij Sorry I missed the ping on this, got buried in all the other gh email. There's no easy solution because the juju network model does not support this use case - the correct way for the charm to ask what it needs to know using network-get cannot supply what's needed. We also do not want to fundamentally break encapsulation by introducing a cross model attribute on the relation that the charm knows about - that's a whole other world of hurt with (bad) far reaching implications beyond this this use case. We discussed again today and will come up with some ideas. We really, really need to extend the juju network model to understand layer 7 concepts etc, but that will be a while away. We agree we need something we can live with until then.

[tlm]

Hey @simskij and @sed-i,

Apologies that I dropped the ball on this one. In any event I did have some discussions a while back and this week to think about this problem more.

I have a bunch of notes already prepped for talks next year to start designing a better solution in Juju for this problem. But might take a bit for us to get a consensus on how we want to model this problem.

In the mean time I think I can see a few ways forward to solve this problem today. Specifically if we utilise Traefik as the ingress for prometheus for all relations this will certainly get us to the end goal.

Without trying to solve this all over GH comments can we jump on a call today or tomorrow for a while and go through what I am thinking? Happy to write up the conclusions from the call here for everyone's consumption.

[benhoyt]

@tlm (When you're back.) Could you or @sed-i please briefly summarize the discussion you had on the call with their team before the break? I gather you didn't find a great workaround apart from something like the _cmr property being proposed.

[tlm]

Hi All,

We had a meeting just before Christmas to discuss this further from the Juju side. My typed notes from the meeting are below. This will act as the input into our discussions that we will start having this week within the Juju team to come up with a designed solution that can be implemented.

Meeting Notes

The COS Canonical team is deploying a set of Operator Charms to offer a complete end to end observability stack. Some of the charmed applications include Prometheus, Grafana and AlertManager. They are deployed as a bundle and all related together in the same model to work together.

It appears that from discussions with the Charm authors that they are using relation data to share certificates between the components to develop a system of trust between the components and also encrypt the over the wire traffic between each component.

All of the components mentioned above are also deployed with a HTTP load balancer in front of them (in this case Traefik). Traefik is used in this instance to provide an ingress point to these charms from components outside of the model. It both reverse proxies the HTTP traffic and also offers a new set of TLS certificates to the connecting client.

The problem starting to arise for this charm is when Juju users wishing to perform cross model relations between components. For this example we will describe the scenario where a Grafana charm has been deployed into a seperate model and needs to be related to Prometheus. At the moment Juju is likely to pass the ip address of the service in Kubernetes to the Grafana charm for its connection (using network-get) . However because of the way the Juju networking model currently works the Grafana charm is still unsure of what server side certificate it should trust during the connection. At the moment the COS bundle being deployed is passing this information over relation data to the Grafana charm but it is currently unable to ascertain the means by which connections are being made to Prometheus and this introduces the issue of which certificate should be provide to Grafana to trust.

It is possible to say as a work around to this problem that the Prometheus charm in this instance could always just return a full list of all the server certificates it knows about and so by whatever means of connection the Grafana charm ends up using it will always trust the server certificate for the HTTP protocol. This still leaves the problem that exists with the Traefik load balancer charm deployed. As part of this bundle design they are wanting to use this layer 7 load balancer to direct all cross model connections through (i.e traffic that is not within the same model).

Currently there exists no way for a charm providing traffic ingress duties for one or more charms to relay this information back to Juju to enrich its network information model. For example lets say a HTTP load balancer charm is providing load balancing and reverse proxying for Prometheus and AlertManager. Juju currently has no way to understand that outside or even inside the model these applications could be reached at https://prometheus.example.com or https://alertmanager.example.com. Instead currently the only information that Juju has is for 10.1.1.1 (being the ip address of the Prometheus service in the Kubernetes cluster). This introduces a few problems as load balancer charms are not always able to get certificates for ip addresses (ACME is an example of this).

One of the proposed methods for solving this problem is to expose a cross model relation flag to the charm. This was briefly discussed and ruled out [1].

At the moment the best way forward that we are currently thinking about is for charms that deploy ingress based resources to the Juju model to offer up this information to Juju so that Jujus networking model can be further enriched and the subsequent information can be disseminated among related Charms. This will probably mean that Juju may need to understand some layer 7 concepts with regards to some protocols (HTTP in this instance).

What we have at least agreed upon thus far is that we don't wish to expose a CMR flag to Charms to identify where a relation is coming from. Adding this is agreed that this is a band aid fix over the real problem described above.

[1] The cross model relation flag was discussed as a means to notify the charm when a relation is happening outside of the model that it is operating in. This has two problems:

1. It is moving logic outside of Juju for how things should be connected together back into the charms themselves.
2. The charms have no context of the large Juju deployment and the networking environment that they are operating in. With this they are unable to enrich or modify the network model to allow or facilitate communications.

[benhoyt]

Thanks @tlm for the thorough write-up.

What we have at least agreed upon thus far is that we don't wish to expose a CMR flag to Charms to identify where a relation is coming from.

Agreed that this isn't going to happen in that form. However, if charms need a hacky workaround the charm could implement it as a stand-alone function is_relation_cross_model(relation: Relation), which as Leon showed is a one-liner.

Further changes to improve ingress handling here will need to be discussed with the Juju team, but closing this specific bug.

[simskij]

@benhoyt @tlm: I believe this issue needs to be reopened, and still disagree strongly with your conclusion. As proven in https://github.com/canonical/cos-lib/pull/30, there is no way for us to roll our solution for detecting this.

I am happy to have a session on this in Hague if you are up for it. But my stance remains: the current decision leaves us with a model where we either have to accept a single point of failure (the ingress) even when there is no actual need, or put an artificial limitation in place that actively blocks cross-model relations for certain relations on certain charms.

Both of these options are, as I'm sure you realize, suboptimal to say the least. For an observability platform, where service availability is critical, this severely impacts the solution's value proposition.

[benhoyt]

@simskij I hear you. I can see how we need something that solves this problem, the Juju team just disagrees that adding a Relation.cross_model property or something similar is how to solve it.

So we need to figure out 1) how to model this for you at the Juju level, and 2) what workaround you can use in the meantime. I just talked to the Juju APAC folks, and they've agreed we should start with a breakout during roadmap week in The Hague (with a follow-on engineering meeting if needed). So I'll schedule that and add you and Leon to start with (add others if you want).

(Side note: does @sed-i's comment here indicate that the workaround might work in this case after all?)