Clarification for unsupported behaviour of chained CCS

[ywangd]

The behaviour of chained CCS, e.g. GET cluster1:cluster2:index/_search, is undefined in Elasticsearch. It works in certain cases but fails sliently in others. Currently we do not have intention to support it. The fact it works in certain cases is accidental and should not be relied upon. We should clarify it in the relevant docs.

Note that a cluster never directly talks to another cluster unless one is configured as remote cluster of the other. This means chained CCS does not violate any intended constraint of network connection rules.

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[ywangd]

Ping @justincr-elastic @gwbrown for awareness. This is not anything like a goal for the new CCx model. But the new model may have a better definition for this behaviour as a by-product.

[gwbrown]

We haven't made an explicit statement, but my suspicion is that we'll completely disallow this kind of "chained query" under the new model in the initial release in order to simplify some of the privilege interaction, but at the same time the new security model would make this kind of chained deployment much easier to codify so I think it could be added without too much effort later - I think the biggest effort in initially supporting it might be building the infrastructure for three-cluster test cases (maybe that's easy now, I'm not up to date on testclusters).

Honestly the biggest question, I think, is: What are the use cases for this? It's neat that we can do this, but I'd want to know more about why people want to do this before building official support for it.

[tvernum]

I don't think it would be easy to support this in the new model. The new RBAC model would define the user's permissions on the originating cluster. Since the originating cluster knows nothing about the 3rd cluster, what permissions would we assign to the user?

[javanna]

My initial take on this is that we should enforce that it's not supported currently in CCS. If it works it's only by accident and we never tested it nor documented it . I would like to see a real usecase where it is needed before we invested effort in it.

[Jaraxal]

Is their proposed architecture to support less flat organizational structures?

It is not always feasible or desired to set up permissions so that every cluster in an organization can talk to every other cluster directly. I see this requirement in the Department of Defense space where organizations are not flat, one-tier structures. There is very much a need for a multi-tier cross-cluster search capability where sub-orgs under an organization's command and control are directly responsible for a subset of other sub-orgs in a nested, multi-tier structure.

[ywangd]

In one of the previous discussions, the user was trying to use this setup to cater network connection constaint, i.e. they wanted to use one cluster in the middle to proxy requests between the other two. But I am not sure whether this networking constraint is directly related to support less flat organisation structure.

[woodywalton]

Agree 💯 with @Jaraxal - large, enterprise-class architectures will absolutely need this sort of multi-level cross cluster chaining, and integrated security across all will be crucial as well.

[bytebilly]

I did some additional research and this is a valid use case in some org types like it was described above.

I'm not sure that we want to target the "network restriction" problem, as it should be addressed via other network solutions and this would sound like a hack. However, the "multi-level orgs" scenario sounds totally legit to me.

Said that, I think that we could invest some time to figure out how it would be possible (and properly done) with the new security model, even if it may be part of a future iteration. Otherwise, this could be a blocker for the possible future removal of the legacy model.

[javanna]

Otherwise, this could be a blocker for the possible future removal of the legacy model.

How could it be if it currently works only in some scenarios and only by accident? We don't have a single test for this and @ywangd pointed out that it does not consistently work.

[bytebilly]

The issue is that today we have users relying on this approach, that apparently is working well for their use case even if it is not intended as an official feature. Feedback from the field is that this is an important feature. If the new model will prevent that behavior, users that need it will just continue using the existing one if they are not able to find a suitable workaround.

That's why I think that we should investigate if we can support it in the new security model. In that case, we can provide an official way to do that, and properly test it. If we figure out that this is not possible, we will clearly state that removing the gray area.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)