search

eclipse-arrowhead / roadmap

Eclipse Public License 2.0
5 stars 9 forks source link

GSoSD Review Discussion Point 5: Should all current kinds of orchestration be supported by the same one orchestration system? #65

Closed jerkerdelsing closed 10 months ago

jerkerdelsing commented 1 year ago

To address the late binding and loose coupled SOA properties "orchestrating" who can/should/shall talk to who is important.

Other similar topics are:

If we limit the orchestration to who can/should/shall talk to who rules which may be based on policies. Such rules can be "owned" by

The current hierarchy developed in Arrowhead is

  1. Operator - Engineering
  2. PlantDescription
  3. Orchestration
  4. Consumer

For interoperability reasons support for both push and pull distribution of orchestration rules should be supported.

The most important usage scenario I see is a top down definition of orchestration rules possibly with plan A,B,C, ... versions to enable full filament of the desired functionality. The various plans can be created by classical engineering to autonomous AI.

For consumer systems who "know" who to talk to we have two possibilities. Upload such desire to either of Orchestration, PlantDescription or Operator. Or that such consumer will be allowed to perform a 100% bottom up orchestration. Which might open for interesting or serious security, safety, functionality, issues.

emanuelpalm commented 1 year ago

As far as I'm concerned, the service orchestrator helps me determine what services I should consume. If I rather decide that on my own, then I use the service discovery service to discover/search by myself for the services I want.

This division of concerns means that the orchestrator can have one simple single push and/or pull service that looks the same no matter what orchestration strategy is used under the hood. It may even be a simple database of mappings between systems and designated services. Other systems, or an operator, may insert these mappings as part of realizing specific orchestration strategies.

I understand that many interesting and sophisticated orchestration strategies have been proposed as part of the Arrowhead project. I agree with the notion that each of them should be a separate system implementation. I would prefer if all of them could implement the same service, if possible.

borditamas commented 1 year ago

AITIA view

As far as I'm concerned, the service orchestrator helps me determine what services I should consume. If I rather decide that on my own, then I use the service discovery service to discover/search by myself for the services I want.

We think that service discovery should be limited to some fundamental core & support system services for an application system. If arrowhead ensures that a service access details are discovered only via the orchestrator service, then a local cloud operator can utilize additional advantages from the different orchestration strategies (intercloud, QoS, matchmaking etc). See discussion point 9. Of course this could be a configuration decision ("limited or unlimited service discovery") of the cloud operator.

This division of concerns means that the orchestrator can have one simple single push and/or pull service that looks the same no matter what orchestration strategy is used under the hood.

From conceptual point of view it would be a good approach for sure. But on the other hand it would require a quite general input and output data model since a more sophisticated strategy can provide more functionalities and may need more inputs. It could be implemented this way we think, but this approach may be more confusing for the users.

Should all current kinds of orchestration be supported by the same one orchestration system?

We agree that different orchestration strategies should be implemented by different systems.

jerkerdelsing commented 1 year ago

Discussion still ongoing

DavidRutqvist commented 1 year ago

From an architecture perspective:

Then each implementation could define what strategy to use for this. The most simple would be just a table of provider consumer mappings as in the first generation of Arrowhead. A more recent implementation would probably have a more sophisticated approach but the service contract should remain the same. Basically, the architecture does not make any distinction. However, Arrowhead as a framework (set of tools) should provide such a sophisticated implementation.

jerkerdelsing commented 1 year ago

From a foundational viewpoint Orchestration shall provide (pull or push) orchestration rules to consumers. The rules an Orchestration system can provide will come from e.g. Management tool, PlantDescription, or other sources. The Orchestration system can store orchestration rules in priority order, Plan A,B,C,....

A to me possible scenario is that most consumers will not have an own understanding of which producer it shall connect to. Thus lifting the orchestration action to a management level using e.g. Management tool, PlantDescription information.

There might be cases (probably rare) where there are strong arguments why the consumer shall know which producer it shall connect to. An interesting question os if and how Arrowhead should be able to track, log and audit consumers acting outside the Orchestration system?

PerOlofsson-Sinetiq commented 1 year ago

Can there be several instances of (various?) Orchestrator systems in an Arrowhead cloud?

PerOlofsson-Sinetiq commented 1 year ago

My opinion concurs with Jerkers idea of having 4 layers of service composition:

Each of the levels must be able to carry out orchestration by themself for resilience reasons. I also think that all levels must be capable of act in an environment where there are multiple instances that compete in achieving their respective goals. Ultimately, the rules of the authorisation systems should limit the competing.

jerkerdelsing commented 1 year ago

Each local cloud has it’s own orchestration system. In principle there can be multiple orchestration systems in a local cloud but that will create synchronisation requirement. Thus the recomendation is to have only one Orchestration and Authorisation system in a local cloud.

The orchestration between local clouds is possible using the Gatekeeper and Gateway microsystems in cooperation with ServiceRegistry and Authorisation/(Authentication) microsystems of other local clouds.

DavidRutqvist commented 1 year ago

Maybe that was not the point, but I see no conflict with my statement and what you both stated above :). As long as there is one API/service type used for the different orchestration scenarios I am happy.

emanuelpalm commented 1 year ago

@borditamas

Of course this could be a configuration decision ("limited or unlimited service discovery") of the cloud operator.

I agree with this. Use of the orchestrator should be optional (while it may be practically required for certain kinds of use cases).

emanuelpalm commented 1 year ago

It seems to me as if the only two things we are not agreeing on are whether or not:

  1. the orchestrator should forward addresses (and tokens?) directly to its consumers, and
  2. whether a single service interface/API is enough for all kinds of orchestration services.
emanuelpalm commented 1 year ago

See also https://github.com/eclipse-arrowhead/roadmap/issues/62#issuecomment-1679606722.

emanuelpalm commented 1 year ago

It was brought up today that there are two more things that should be discussed:

  1. Auditing. If the orchestrator is not used, then how do we monitor what the systems are doing in the cloud?
  2. Metadata in orchestration rules. How do I tell a consumer how they should use the services the orchestrator refers to? Read? Write? Something else?
jerkerdelsing commented 1 year ago

17/8 meeting. JD to summarise a proposal for the v5.0 specification.

jerkerdelsing commented 10 months ago

Introduced into the GSoSD by Jerker