apache / camel-kamelets

Apache Camel Kamelets Repository
https://camel.apache.org
Apache License 2.0
52 stars 71 forks source link

Support DataRef cloudevent extension #1872

Open matzew opened 5 months ago

matzew commented 5 months ago

In the CloudEvent spec, there is the dataref extension defined: https://github.com/cloudevents/spec/blob/main/cloudevents/extensions/dataref.md

It would be nice to have a way to make use of this, especially to avoid sending large data, e.g. in sources that shuffle data, like from AWS for example:

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: aws-s3-source-pipe
  annotations:
    trait.camel.apache.org/camel.runtime-version: 3.6.0
spec:
  integration:
    dependencies:
    - "camel:cloudevents"
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: aws-s3-source
    properties:
      .....
oscerd commented 4 months ago

Supposing we are going to support multiple components with cloud events this require a lot of work: for s3 for example, you can decide to exclude the body from the payload, by specifying a parameter in the Kamelet configuration, the problem is much more related to the fact you don't have a way to directly point to the S3 object: if it's a public object we could build the URL, if it's not we should create a pre-signed URL which require some work but also some API call and some cost.

Doing for all the CE supported components is not easy and require a lot of time. So not sure this could be done without some coordinated effort.