Improving coercing dynamic properties to specific types

[anna-geller]

Feature description

This example requires standing up the pulsar docker-compose stack + the pulsar plugin package. This will throw an error as the input to the from property is a string and so the plugin thinks it is a URI pointing to an internal file. (using | does the same thing)

id: hello-world
namespace: test

variables:
  dict: '{ "a": 2, "b": "hello", "c": "", "d": 45 }'

tasks:
  - id: hello
    type: io.kestra.plugin.pulsar.Produce
    uri: pulsar://localhost:26650
    serializer: JSON
    topic: test_kestra
    from: "{{ vars.dict }}"

However, if you pass a map directly, it works ok:

id: hello-world
namespace: test

variables:
  dict: '{ "a": 2, "b": "hello", "c": "", "d": 45 }'

tasks:
  - id: hello
    type: io.kestra.plugin.pulsar.Produce
    uri: pulsar://localhost:26650
    serializer: JSON
    topic: test_kestra
    from: { "a": 2, "b": "hello", "c": "", "d": 45 }

The issue is opened here rather than on Pulsar as it's a more generic issue, e.g. the same in Kafka plugin:

- id: avro_to_kafka
    type: io.kestra.plugin.kafka.Produce
    from: {{ json('[1, 2, 3]') }} <- This gives an error and we have to do "{{ json('[1, 2, 3]') }}" which becomes a string
    .....

passing JSON-type input leads to the same issue:

id: hello-world
namespace: test

inputs:
  - name: dict
    type: JSON
    defaults: '{ "a": 2, "b": "hello", "c": "", "d": 45 }'

tasks:
  - id: hello
    type: io.kestra.plugin.pulsar.Produce
    uri: pulsar://localhost:26650
    serializer: JSON
    topic: test_kestra
    from: "{{ inputs.dict }}"

atm, it doesn't matter what the output of the expression is, it always gets converted to a string because we have to escape the {{ with " or pipe |

Ideas worth exploring

We may consider using YAML tags e.g. from: !!map "{{ inputs.dict }}" to let users cast some dynamically rendered values to a specific data type expected by a plugin:

id: hello-world
namespace: test

inputs:
  - name: dict
    type: JSON
    defaults: '{ "a": 2, "b": "hello", "c": "", "d": 45 }'

tasks:
  - id: print
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ inputs.dict }}"

  - id: message
    type: io.kestra.plugin.pulsar.Produce
    uri: pulsar://localhost:26650
    serializer: JSON
    topic: test_kestra
    from: !!map "{{ inputs.dict }}"

Alternatively, we need to reconsider how some plugins render dynamic properties to enforce coercing types

The https://github.com/kestra-io/kestra/issues/2962 might be one way to handle such use cases, but more as a workaround

Alternative - pipe operator to a given type

id: hello-world
namespace: test

inputs:
  - name: dict
    type: JSON
    defaults: '{ "a": 2, "b": "hello", "c": "", "d": 45 }'

tasks:
  - id: print
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ inputs.dict }}"

  - id: message
    type: io.kestra.plugin.pulsar.Produce
    uri: pulsar://localhost:26650
    serializer: JSON
    topic: test_kestra
    from: "{{ inputs.dict | map }}"

[anna-geller]

We need to change the way we render variables to support more than only strings

[loicmathieu]

Fixed by #4683