Generic resource support in Pipe

[lburgazzoli]

Context

I did some experiments with Crossplane which allows to orchestrate resources across cloud providers by creating custom resources, as example wit the following CRs:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: foo-my-bucket
spec:
  forProvider:
    acl: private
    locationConstraint: eu-south-1
  writeConnectionSecretToRef:
    name: my-s3
    namespace: default

apiVersion: sqs.aws.crossplane.io/v1beta1
kind: Queue
metadata:
  name: foo-my-sqs
spec:
  forProvider:
    region: eu-south-1
  writeConnectionSecretToRef:
    name: my-sqs
    namespace: default

Will result in an SQS queue and S3 bucket being created on AWS and the nice thing is that the status sub resource of those resources, reports information about the resource, as example, the status for the queue defined above is:

  status:
    atProvider:
      arn: arn:aws:sqs:eu-south-1:xxx:foo-my-sqs
      url: https://sqs.eu-south-1.amazonaws.com/xxx/foo-my-sqs

Additional information may be projected to the secret defined by the writeConnectionSecretToRef property.

Problem

With https://github.com/apache/camel-k/issues/2625, we want to rename KameletBinding to Binding as a binding is a generic concept that goes beyond kamelets, however the underlying machinery could still be leveraging a kamelet to perform the actual wiring but the issue is that as today we have to introduce ad hoc support for any new resource we want to support which is unmaintainable in the long run.

Proposed solution

Crossplane has a concept of composite resources which allows to compose a number of managed resources so that you can build your own APIs without introducing any new controller/operator. One part of the composition is about to bind a field of the newly crafted API to the underling managed resource and a similar approach can be used to be able to dynamically configure a kamelet from an arbitrary CR

As example, assuming we write a binding to wire the resources created in the example above:

apiVersion: camel.apache.com/v2
kind: Binding
metadata:
  name: sqs-to-s3
spec:
  source:
    ref:
      apiVersion: sqs.aws.crossplane.io/v1beta1
      kind: Queue
      name: my-queue
  sink:
    ref:
      apiVersion: s3.aws.crossplane.io/v1beta1
      kind: Bucket
      name: my-bucket

We can define a new resource (find a better name):

apiVersion: camel.apache.com/v2
kind: KameletResourceBinding
metadata:
  name: crossplane-sqs
spec:
  selectors:
    - apiVersion: sqs.aws.crossplane.io/v1beta1
      kind: Queue
  kamelets:
    - apiVersion: camel.apache.org/v2
      kind: Kamelet
      name: aws-sqs-source
    - apiVersion: camel.apache.org/v2
      kind: Kamelet
      name: aws-sqs-sink
  propertiesRef:
    - name: "queueNameOrArn"
      fromFieldPath: status.atProvider.arn
    - name: "region"
      fromFieldPath: spec.forProvider.region
    - name: "accessKey"
      fromSecretKeyRef: 
        nameRef: spec.forProviderwriteConnectionSecretToRef
        key: foo           

When camel-k reconciles the binding:

• it looks up for any KameletResourceBinding matching the configured resources
• it extracts the relevant information from the resources
• it generates the route and secrets according

NOTE: secrets management is still to be investigated.

[lburgazzoli]

@christophd @astefanutti @squakez @oscerd @valdar what do younthink ?

[oscerd]

It looks promising.

[johnpoth]

Probably a silly question but isn't this what the Service Binding Operator was suppose to do ? Or is this just a better way of achieving binding ? Thanks!

[lburgazzoli]

Probably a silly question but isn't this what the Service Binding Operator was suppose to do ? Or is this just a better way of achieving binding ? Thanks!

Right is it quite similar and I was thinking about the SBO when writing the issue, however I found that it is not that easy to use SBO for generic resources and in the context of KameletBinding as you may need to have knowledge about the resource that exposes the information and the consumer and with my probably outdated knowledge of SBO what it would be required is:

• to annotate the Binding and Bucket resource
• to annotate the kamelet binding for the correct mapping

So the main goal here is to make it possible for kamelet developers to define how the mapping between resources should be done, regardless of how the binding is performed so i.e. the camel-k operator could generate any SBO metadata/resource to materialize the binding or it can go it's way if the SBO operator is not present.

[lburgazzoli]

Thinking a little bit more, this could also be useful for the other way i.e. how to bind kamelet properties to i.e. KEDA in a customizable way (as today such info is part of property definitions in the kamelet)

[github-actions[bot]]

This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!

[cmoulliard]

Probably a silly question but isn't this what the Service Binding Operator was suppose to do ? Or is this just a better way of achieving binding ? Thanks!

FYI, we are working to improve the Service Binding concept which is not so easy to be used (as mentioned by Luca) due to lack of discovering capability, registration of services, easy way to declare the parameters to bind, ...

This is why we are working on a new project called Primaza aimed to improve the service binding, which could be designed (perhaps) top of dapr, .... in order to benefit from some of their features such as:

• Service invocation: discovery, etc
• Secret management (to better secure the data of a connection by example):
• Binding: To be investigated !

[oscerd]

This is still relevant.