Cloud (?!) fails on finding and providing requested service

[strohbert]

We posted this issue in the slack (https://arrowhead-dev.slack.com/archives/CBBMWTZNZ/p1621001024046300) but did not get any response. Therefore I opened an issue here directly in the repo.

We are trying to start up the kalix echo-cloud example (https://github.com/arrowhead-f/arrowhead-kalix-examples/tree/master/echo-cloud). We can successfully run the provider, but the cloud (?!) fails on finding and providing the service (see exception below). We are running version 4.2.0 of arrowhead (https://github.com/arrowhead-f/core-java-spring) and we did not alter the kalix example in any way. The registration of both, the provider and the consumer seems to work just fine as the entries in the database suggest. We are using certificates based on the testcloud2.aitia.arrowhead.eu and the provider starts flawlessly but the consumer fails on sending the request (line 83 of EchoConsumer.java .flatMap(consumer -> consumer.send(new HttpConsumerRequest() ). We are also unsure about the line INFO: HTTP/JSON cloud plugin resolved orchestration service at /127.0.0.1:8441 - as we expect something like localhost:8441 as endpoint.

...
Consumer service is being provided ...
Mai 14, 2021 3:36:52 PM se.arkalix.core.plugin.HttpJsonCloudPlugin$Attached requestOrchestration
INFO: HTTP/JSON cloud plugin connecting to "orchestrator" system ...
Mai 14, 2021 3:36:54 PM se.arkalix.core.plugin.HttpJsonCloudPlugin$Attached lambda$requestOrchestration$17
INFO: HTTP/JSON cloud plugin resolved orchestration service at /127.0.0.1:8441
GET /example/pings/32 failed:
se.arkalix.query.ServiceNotFoundException: No service with the following properties could be resolved: name=Optional[kalix-example-provider-service], isSecure=true, encodings=[JSON], transports=[HTTP]
 at se.arkalix.query.ServiceQuery.lambda$resolveOne$4(ServiceQuery.java:451)

Any suggestions or hint, what might be the problem?

[emanuelpalm]

Hey @strohbert! Nice reading that you have been giving Kalix a go.

Are you using the docker-compose.yml file to run the example? It automatically sets up all the authorization and orchestration rules necessary for the Echo Consumer to lookup and access the Echo Provider. The setting up is performed using the Cloud Configurator, which is a custom-made system meant to make it easier to configure local clouds. It is only meant to exist until an official solution for making such convenient configurations becomes available. It requires a configuration file to work. An example can be seen here.

The reason I believe missing authorization and orchestration rules to be the issue is that you get a ServiceNotFoundException, which is typical when proper rules are not configured. I hope this helps!

[awoSYS]

Hi @emanuelpalm! I'm working together with @strohbert trying to get this example to work and understand it. Thanks for your support! We were trying to simply start the two systems (echo provider and consumer) by running the respective jars and then got the error. From your post I understand that this is not going to work, since the cloud configurator system is supposed to do all the hard work setting up the AH rules such that provider and consumer are able to find each other. However, I'm still having trouble to understand that configurator system.

• You mentioned, the cloud config system is there to setup the rules only for this complex example. Would this code have to be entirely rewritten/changed for a productive use case?
• Is there any documentation that helps to understand what that configurator system is doing? It's really a lot of code ...
• Is there any simpler example or what's your recommended way to get started with Kalix and AHFW in a minimal scenario? Thanks again for your help!

[emanuelpalm]

@awoSYS @strohbert I assume you tried the cloud-standalone example project first? Did that work out for you?

The difference between the standalone and cloud examples is that the former consists solely of the two systems (i.e. Java applications) provided in the example. The latter provides three systems (echo provider, echo consumer and configurator), and assumes that you have three more running (a Service Registry system, an Authorization system and an Orchestrator system). The reason for the increased complexity of the second example is that the extra systems allows for the echo consumer and provider to dynamically find each other and exchange messages, without prior knowledge about each other IP addresses or other details.

All systems part of the same local cloud (a cluster of systems all managed by the same operator) must have X.509 certificates issued by the same cloud certificate holder. All certificates must conform to the Eclipse Arrowhead certificate profile (which is not yet published/formalized, but a brief description of its current implementation can be read here).

If dynamic service discovery is desired (as in the echo-cloud example), all systems in the local cloud must know the IP address of the Service Registry system, and the port (and other details) through which its Service Discovery service can be reached. The service registry, authorization and orchestrator systems must, of course, all be running. In addition to knowing the IP address of the Service Registry system, there must be authorization rules and orchestration rules setup in the Authorization and Orchestrator systems. This is what the configurator system in the example does. It registers the rules necessary for dynamic service resolution between the echo consumer and provider.

The documentation situation currently leaves quite a lot to be desired. The official one is here. I have written quite some documentation for the Kalix library here. Important sections to read are [1] and [2].

Just tell me if you have any more questions. I'll try to answer as soon as time permits.

[awoSYS]

Thanks @emanuelpalm for that much information!

I understand that the standalone example is only supposed to show how the setup would look like with producer and consumer being wired via hardcoded addresses. I ran it and it worked:

Logs

``` DELETE /example/runtime result: 204 No Content GET /example/pings/32 result: 200 OK {"ping":"pong","id":"32","timestamp":"2021-06-21T14:00:31.642509Z"} POST /example/pings result: Ping{ping='pong!', id='null', timestamp=null} Done! ```

The cloud example also seems to work now:

Echo consumer logs

``` DELETE /example/runtime result: 204 No Content GET /example/pings/32 result: Ping{ping='pong', id='32', timestamp=2021-06-21T13:50:56.727118Z} ```

(Our earlier problems were most likely caused Linux-Windows character encoding incompatibility --> Checking out the repo on Windows made the commands in docker-compose.yml unreadable for the Linux containers).

However, I've got a couple of questions left:

• I cannot open the Swagger UIs of the core services, even though I've added the sysop.p12 and arrowhead.eu certificates to the browser and set the latter to be allowed to trust websites. As long as the containers are running, the website (e.g. https://localhost:8443) just never stops loading, no error screen or so. Do you have any idea, what might be wrong?
• I don't understand, why the configurator system is needed. In a productive scenario, wouldn't each provider istelf be responsible of registering its service to the cloud to make it discoverable for consumers? Or am I missing something here?
• In the configurator logs, I found that it basically performs the following 4 actions:

  2021-06-21 13:30:50 INFO Created service entry echo_provider/kalix-example-provider-service
  2021-06-21 13:30:51 INFO Created service entry echo_consumer/kalix-example-consumer-dummy
  2021-06-21 13:30:54 INFO Created authorization rule CfConsumptionRule{consumer='echo_consumer', services=[kalix-example-provider-service], providers=[echo_provider]}
  2021-06-21 13:30:56 INFO Created orchestration rules [CfConsumptionRule{consumer='echo_consumer', services=[kalix-example-provider-service], providers=[echo_provider]}]

  • I understand the following:
  • action 1 is to register the provider's service to make it discoverable. It would normally be done by the provider.
  • action 2 is required for the consumer to be able to consume any service at all. It would normally be done by the consumer.
  • action 3/4 are for the consumer to be allowed to consume that specific service.
  • My questions:
  • Who would do actions 3/4 in a productive scenario? Provider or consumer?
  • Why are two actions required to allow the consumer to consume the service? What's the difference between the orchestration rule and the authorization rule?

Thanks for your support!

[emanuelpalm]

Nice that you are making progress!

• Opening the Swagger UIs.
  • This can be a tough one, as browser support for client-side TLS support has proven to be quite limited/buggy, nomatter if using Firefox or Chrome. If you open the developer console, open the network view, make sure that messages are preserved between page reloads and then reload the page again, I'm quite sure you will see some kind of TLS handshake error. Try removing the sysop.p12 certificates from the browser, add them again and make sure that the arrowhead.eu certificate is trusted to identify websites. If that doesn't work, it may help if you also the add the truststore.p12 to the browser.
• Why the configurator system is needed.
  • When a system tries to consume/use any of the services you provide, you need to determine if the system is allowed to do so. The way Arrowhead is currently designed, there are two ways of determining this. The first way is currently called certificate authorization and means that the consuming system is able to present a certificate satisfying some criteria (usually being issued by the same local cloud certificate as the service provider, and sometimes also having a certain certificate CN (Common Name)). This is the method used in the echo-standalone example. The other way is called token authorization and requires service consumers to include a token with every request that has been issued by an Authorization system. This is the method used in the echo-cloud example. The second method requires that the Authorization system knows what systems are allowed to consume what services. The configurator in the echo-cloud example creates such authorization rules.
  • When a system wants to consume/use a certain service, it may be the case that multiple systems provide that service, and that our system is allowed to consume more than one of them. Which should it access? This is where the Orchestrator system comes in handy. It maintains, what could be thought of as, priority rules for service access. When a system starts up, it will typically first register itself via the Service registry, ask the Service registry for the IP address/port/other details of the Orchestrator system, and then ask the Orchestrator about what services to consume. The Orchestrator system answers with IP addresses/ports/other details required to access those services, including any authorization tokens. However, if the Orchestrator is to be able to answer, it must have a set of relevant orchestration/priority rules. These are setup by the configurator system in the echo-cloud example.
• Understanding the configurator log output.
  • Due to the way the Authorization and Orchestrator systems are designed, they cannot store authorization or orchestration rules without all systems and services associated with those rules being registered in the service registry. As the echo_provider and echo_consumer systems are designed to immediately try to contact each other (and not retry if they fail), all of their authorization and orchestration rules must exist before they start. For this reason, the configurator registers them before they get the chance to themselves. Kalix will likely be made to handle this more gracefully in the future.
• Who would take responsibility for creating authorization and orchestration rules in a production scenario?
  • The operator (the person operating the local cloud) would create them. However, the methods that currently exist for doing this are all rather limited. There is something called the Management Tool that could be used. You can also use the Swagger interfaces you already tried to access. We are working on ways to make this much more intuitive, so you can expect this to improve over time.

Any more questions? Did I miss anything?

[awoSYS]

Thanks @emanuelpalm, your explanations already helped me a lot understanding what's going on! If you're not yet exhausted of my asking though, I'd be happy about even more input from you!

I didn't manage to open the swagger UIs, neither using Chromium, nor Firefox. I'll skip that for now.

About the configurator system:

• I understand that in order to be able to create authorization/orchestration rules, it is necessary for all consumed services and consuming systems to be registered in advance. Do you know, if this is still valid/required for the most recent version of the AHFW? I'm still struggeling to understand the benefits of that concept...
• Does every service get registered twice, once by the configurator to create the orchestration/authorization rule and once by the app system itself to actually provide the service?
• Why is the configuration logic (package se.arkalix.util.config) not part of the Kalix library? Even though in a productive scenario there is the opportunity to manually configure such rules via any sort of UI, isn't it always a good idea to automate this, as you've done it in this example?

About security and authorization:

• I understand from your explanations and the Kalix docs, that there are multiple authorization paradigms available (certificate, token-based, ...). The official AHFW GitHub docs only mention the token-based approach. Do you know, where I can learn more about the available concepts?
• In any of the secure approaches, each system has to have a certificate, that is derived from the local cloud certificate, right?
• Is it fair to say: When using the Kalix library, the provider/consumer code is equal, no matter if AccessPolicy.token() or AccessPolicy.cloud() is being used (apart from specifying this once in the provider code). The actual authorization mechanisms performed when a consumer knoks at the provider's door, are completely transparent to the developer.
• The truststore used for all (core and application) systems in the echo-cloud example only contains the master certificate (arrowhead.eu). Does that mean, all systems trust each other system that has an identity derived from arrowhead.eu, regardless of the company and cloud certificates further down the trust chain?

Sorry for the amount of text! Feel free to ignore what you don't fancy to answer ;) And thanks for your support again!

[emanuelpalm]

Configurator system

1. It is still something that is valid for the last verion of AHFW, as far as I know. I may be wrong though, as I do know that this is something that is being discussed and worked on. My impression is that it will not be fixed until AHFW version 5, but may be that I'm not up to speed with the latest developments.
2. Well, the Kalix cloud plugin will automatically attempt to register the services provided by its systems, which means that in the echo-cloud demo, registration occurs twice. There is nothing that requires this behavior, however. I will likely revisit the solution and try to improve it in future versions of Kalix.
3. Because how this is to be solved in Arrowhead is still under consideration. My configurator is not to be regarded as more than an interim until a consensus has been reached and a workable solution produced.

Security and Authorization

1. The token-based approach is strongly recommended (as far as I know) for all systems that are not essential core systems provided by the Eclipse Arrowhead project, for which reason the documentation focuses on it. The certificate-only approach is something the developers of the original Arrowhead core systems figured out they had to have in order to be able to start up a functional local cloud that can then offer token-based authorization to the rest of its systems. More access control mechanisms are being discussed, such as being able to use different kinds of tokens and having different kinds of certificate-only criteria.
2. A certificate issued by the local cloud certificate is required for every system to be part of that local cloud, irrespective of what access control policy (i.e. secure mode) is used. Yes.
3. Yes. Developers should be aware, however, that the timing, performance and reliance implications are different.
4. The reason for a truststore being required at all is because of how TLS operates. TLS requires that communicating systems agree on a single issuer that both systems trust. That certificate could just as well have been extracted from the key store of each system (as it also contains the same certificate, which I'm sure you've noticed). By having a separate truststore, however, I leave room for being able to use TLS to connect to non-Arrowhead systems. I may change the behavior of Kalix to extract the truststore from its keystore if no truststore is specified. We'll see.

I hope this helped!

[awoSYS]

Thanks @emanuelpalm, I understand ArKalix and the AHFW much better now!!

[emanuelpalm]

@awoSYS No problem! If you believe your questions to be answered at this point. Please close this issue. Thank you!

[awoSYS]

@strohbert imho you can close this issue.