The Yaml Parser diverges for Kamelets and Integration Resources

andy-d2019 commented 5 months ago

What happened?

While the following Yaml Syntax:

          - constructors:
            '0': '{{influx-url}}'
            '1': '{{influx-username}}'
            '2': '{{influx-password}}'
          factoryBean: org.influxdb.InfluxDBFactory
          factoryMethod: connect
          name: influxClient
          type: org.influxdb.InfluxDB

Works perfectly fine when applied in a Camel Integration resource directly. If it is used in a Kamelet, that is referenced in a Pipe, the Integration that is build and executed on the cluster runs into an unsupported field exception.

The unsupported Field Exception will appear with any of the fields factoryBean, factoryMethod and constructors, depending on the order they are used in.

Relevant log output

Caused by: Unsupported field: constructors
 in file:/etc/camel/kamelets/kamelets-bundle-influx-action-binding-001/write-influx-action.kamelet.yaml, line 58, column 9:
            "0": '{{influx-url}}'
            ^

    at org.apache.camel.dsl.yaml.common.YamlDeserializerBase.handleUnknownProperty(YamlDeserializerBase.java:133)
    at org.apache.camel.dsl.yaml.common.YamlDeserializerBase.setProperties(YamlDeserializerBase.java:127)
    at org.apache.camel.dsl.yaml.common.YamlDeserializerBase.construct(YamlDeserializerBase.java:66)
    at org.apache.camel.dsl.yaml.common.YamlDeserializationContext.lambda$resolve$0(YamlDeserializationContext.java:157)
    at org.apache.camel.dsl.yaml.common.YamlDeserializerSupport.asType(YamlDeserializerSupport.java:344)

Camel K version

2.2.0 (Camel 4.2.0, Apache Camel K Runtime 3.6.0)

squakez commented 5 months ago

Do you mind providing a reproducer with basic sources? It would help troubleshooting. Also, mind that Camel K 2.2.0 is running by default with Camel Quarkus 3.2.0 runtime.

andy-d2019 commented 4 months ago

Sure, sorry for the late reply.

This is a basic correctly working integration:

---
# yamllint disable-line rule:line-length
# camel-k: language=yaml dependency="camel:influxdb" trait=camel.runtime-version="3.6.0"
- beans:
    - type: org.influxdb.InfluxDB
      name: influxClient
      factoryBean: org.influxdb.InfluxDBFactory
      factoryMethod: connect
      constructors:
        0: '{{influx-url}}'
        1: '{{influx-username}}'
        2: '{{influx-password}}'

- route:
    id: influx-reader
    description: >
      This route reads data from influxdb for testing purposes.
    from:
      uri: timer://read?repeatCount=1
      steps:
        - setHeader:
            id: setInfluxQuery
            description: >
              Set the header 'influxQuery' to the query that should be executed.
            name: camelInfluxDB.query
            simple: "SELECT * FROM assets"
        - to:
            id: retrieveAssetFromInfluxDb
            description: >
              Query the influxdb for assets.
            uri: 'influxdb:influxClient'
            parameters:
              databaseName: "{{influx-database}}"
              retentionPolicy: "{{influx-retentionpolicy}}"
              operation: 'query'
        - log:
            id: logSuccess
            loggingLevel: 'INFO'
            message: '$simple{id}: Data has been retrieved from the influxdb.'

And now two basic Kamelets and Pipe that lead to the aforementioned error:

apiVersion: camel.apache.org/v1
kind: Kamelet
metadata:
  name: influx-query-action
  annotations:
    trait.camel.apache.org/camel.runtime-version: 3.6.0
    camel.apache.org/kamelet.support.level: "Stable"
    camel.apache.org/catalog.version: "4.2.0"
    camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pg0KPCEtLSBHZW5lcmF0b3I6IEFkb2JlIElsbHVzdHJhdG9yIDE2LjAuMCwgU1ZHIEV4cG9ydCBQbHVnLUluIC4gU1ZHIFZlcnNpb246IDYuMDAgQnVpbGQgMCkgIC0tPg0KPCFET0NUWVBFIHN2ZyBQVUJMSUMgIi0vL1czQy8vRFREIFNWRyAxLjEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4NCjxzdmcgdmVyc2lvbj0iMS4xIiBpZD0iQ2FwYV8xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB4PSIwcHgiIHk9IjBweCINCgkgd2lkdGg9IjUxMnB4IiBoZWlnaHQ9IjUxMnB4IiB2aWV3Qm94PSIwIDAgNTEyIDUxMiIgc3R5bGU9ImVuYWJsZS1iYWNrZ3JvdW5kOm5ldyAwIDAgNTEyIDUxMjsiIHhtbDpzcGFjZT0icHJlc2VydmUiPg0KPGc+DQoJPHBhdGggZD0iTTQ0OCwwSDY0QzQ2LjMyOCwwLDMyLDE0LjMxMywzMiwzMnY0NDhjMCwxNy42ODgsMTQuMzI4LDMyLDMyLDMyaDM4NGMxNy42ODgsMCwzMi0xNC4zMTIsMzItMzJWMzINCgkJQzQ4MCwxNC4zMTMsNDY1LjY4OCwwLDQ0OCwweiBNNjQsNDgwVjEyOGg4MHY2NEg5NnYxNmg0OHY0OEg5NnYxNmg0OHY0OEg5NnYxNmg0OHY0OEg5NnYxNmg0OHY4MEg2NHogTTQ0OCw0ODBIMTYwdi04MGgyNTZ2LTE2DQoJCUgxNjB2LTQ4aDI1NnYtMTZIMTYwdi00OGgyNTZ2LTE2SDE2MHYtNDhoMjU2di0xNkgxNjB2LTY0aDI4OFY0ODB6Ii8+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8L3N2Zz4NCg=="
    camel.apache.org/provider: "Apache Software Foundation"
    camel.apache.org/kamelet.group: „company“
  labels:
    camel.apache.org/kamelet.type: "action"
    # The name of the application.
    app.kubernetes.io/name: influx-action
    # The component within the architecture.
    app.kubernetes.io/component: test
    # The name of the higher level application this application is part of.
    app.kubernetes.io/part-of: test
spec:
  dependencies:
  - "mvn:org.apache.camel:camel-influxdb:4.2.0"
  dataTypes:
     out:
      default: json
      types:
        json:
          mediaType: application/json
  definition:
    title: "Influx-Query-Action"
    description: |-
     This action retrieves assets from an influxDB.
    type: object
    required:
        - influx-url
        - influx-username
        - influx-password
        - influx-database
        - influx-retentionpolicy
    properties:
        influx-url:
          title: Address to locate the influxdb
          example: http://localhost:8086
          type: string
        influx-database:
          title: The database to query from
          example: primary
          type: string
        influx-username:
          title: The username used to access the influx db
          example: testuser
          type: string
        influx-password:
          title: The token used to access the influx db
          example: asdwqe23
          type: string
        influx-retentionpolicy:
          title: The retention policy applied to the influx db
          example: autogen
          type: string
  template:
    beans: 
      - type: org.influxdb.InfluxDB
        name: influxClient
        factoryBean: org.influxdb.InfluxDBFactory
        factoryMethod: connect
        constructors:
          0: '{{influx-url}}'
          1: '{{influx-username}}'
          2: '{{influx-password}}'

    from:
      uri: "kamelet:source"
      steps:
         - to:
            id: queryInfluxDb
            description: >
              Query the influx database.
            uri: influxdb:influxClient
            parameters:
              query: "$simple{body}"
              databaseName: "{{influx-database}}"
              retentionPolicy: "{{influx-retentionpolicy}}"
              operation: QUERY
         - to: "log:info"

apiVersion: camel.apache.org/v1
kind: Kamelet
metadata:
  name: influx-generator-source
  labels:
    camel.apache.org/kamelet.type: "source"
  annotations:
    trait.camel.apache.org/logging.level: DEBUG
    camel.apache.org/kamelet.support.level: "Stable"
    camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNTYgMzA4LjIzNDAxIj48dGl0bGU+YXdzLWtpbmVzaXM8L3RpdGxlPjxwYXRoIGQ9Ik0wLDE3Mi4wODdsMTI3Ljc1NCw1OC44MSwxMjcuNzUyLTU4LjgxLTEyNy43NTItNS4yOTNaIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDAuMDAwMDUpIiBmaWxsPSIjZmNiZjkyIi8+PHBhdGggZD0iTTEyOC4xNDcsMCwuMDU5LDYzLjg4MXY5MC4xMzZIMTUzLjY0OFYxMi43NTFaIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDAuMDAwMDUpIiBmaWxsPSIjOWQ1MDI1Ii8+PHBhdGggZD0iTS4wNTksMjE3LjU1OWwxMjguMTYyLDkwLjY3NUwyNTYsMjE3LjU1OSwxMjcuOTQ1LDE5OC45MjZaIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDAuMDAwMDUpIiBmaWxsPSIjZmNiZjkyIi8+PHBhdGggZD0iTTEyOC4xNDYsMTU0LjAxN2g2Ny41NzdWNTcuODM2TDE3NS45OSw0OS45NDMsMTI4LjE0Niw2My44OThaIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDAuMDAwMDUpIiBmaWxsPSIjOWQ1MDI1Ii8+PHBhdGggZD0iTTE3NS45OSwxNTQuMDE3aDUyLjIzM1Y5MS42MzJsLTE0Ljk0LTQuNDgxLTM3LjI5Myw2LjMzWiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAwLjAwMDA1KSIgZmlsbD0iIzlkNTAyNSIvPjxwYXRoIGQ9Ik0yMTMuMjgyLDgyLjI2djcxLjc1N2g0Mi4yMjRMMjU2LDgxLjk0MWwtMTIuODI2LTUuMTI0WiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAwLjAwMDA1KSIgZmlsbD0iIzlkNTAyNSIvPjxwYXRoIGQ9Ik0xMjguMTQ3LDBWMTU0LjAxN2gyNS41VjEyLjc1MVoiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiNmNjg1MzQiLz48cGF0aCBkPSJNMTk1LjcyNCw1Ny44MzZsLTE5LjczMy03Ljg5NFYxNTQuMDE3aDE5LjczMloiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiNmNjg1MzQiLz48cGF0aCBkPSJNMjI4LjIyNCw5MS42MzJsLTE0Ljk0MS00LjQ4djY2Ljg2NWgxNC45NFoiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiNmNjg1MzQiLz48cGF0aCBkPSJNMjQzLjE3NCwxNTQuMDE3SDI1NlY4MS45NDFsLTEyLjgyNi01LjEyNFoiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiNmNjg1MzQiLz48cGF0aCBkPSJNMTI3Ljc1NCwxODQuODYzdjQ2LjAzM2wxMjcuNzUyLTMxLjg0NFYxNzIuMDg3WiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAwLjAwMDA1KSIgZmlsbD0iI2Y2ODUzNCIvPjxwYXRoIGQ9Ik0xMjcuNzU0LDI2Mi43ODF2NDUuNDUzTDI1NiwyNDQuMTE0VjIxNy41NloiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiNmNjg1MzQiLz48cGF0aCBkPSJNLjA1OSwyNDQuMzlsMTI3LjY5NSw2My44NDRWMjYyLjQ0OEwuMDU4LDIxNy41NThaIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwIDAuMDAwMDUpIiBmaWxsPSIjOWQ1MDI1Ii8+PHBhdGggZD0iTTAsMTk5LjA1MWwxMjcuNzU0LDMxLjg0NVYxODQuODYyTDAsMTcyLjA4NloiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMC4wMDAwNSkiIGZpbGw9IiM5ZDUwMjUiLz48L3N2Zz4="
    camel.apache.org/provider: "test Gmbh"
    camel.apache.org/kamelet.group: "test"
spec:
  dependencies:
  - "camel:kamelet"
  - "camel:core"
  - "camel:log"
  definition:
    title: "Report-generator"
    description: "Sends a test input to another kamelet"
    required:
      - query
    properties:
      query:
        title: queries
        description: The query map to split up into separate influx queries
        type: string
        example: '"SELECT * from Hosts"'
  dataTypes:
    out:
      default: text
      types:
        text:
          mediaType: text/plain
  template:
    from:
      uri: timer:tick
      steps:
        - setBody:
            constant: "{{query}}"
        - to: "log:info"

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: influx-action-binding
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: influx-generator-source
    properties:
      query: "from(bucket:\"company\") |> range(start: 0)"
  steps:
  - ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: influx-query-action
    properties:
      influx-database: test
      influx-url: "http://my-influxdb.test.svc.cluster.local:8086"
      influx-username: "test"
      influx-password: "test12345"
      influx-retentionpolicy: "autogen"
  sink:
     uri: http://influx-service.test/queryresult

The integration for the influx-query-action will build without error and will be executed, but runtime errors due to unsupported fields will occur. Setting up an influxDB for reproduction should not be necessary, as the given runtime error occurs before the specified Bean is constructed.

squakez commented 4 months ago

I think it could be because in the yaml spec [1], the two objects are really different. One is type TemplatedRouteBeanDefinition and the other is type RouteTemplateBeanDefinition. I am not sure if the expected format and related implementation is different or it is some possible bug on Camel. @davsclaus @oscerd @lburgazzoli wdyt?

[1] https://github.com/apache/camel/blob/main/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camelYamlDsl.json

squakez commented 3 months ago

@lburgazzoli @davsclaus @oscerd bump. Please, confirm if this could be a bug on the core or something else we may need to fix in Camel K. Thanks in advance.

davsclaus commented 2 months ago

https://issues.apache.org/jira/browse/CAMEL-20514

squakez commented 2 months ago

Will be fixed in Camel > 4.6.0 runtimes

apache / camel-k