search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.09k stars 814 forks source link

Support remote templates #1838

Open natenho opened 1 year ago

natenho commented 1 year ago

Hi!

An idea to improve Templating would be to allow referencing local and remote templates, something like this:

input:
  template:
    source: https://raw.githubusercontent.com/benthosdev/benthos/main/config/template_examples/processor_log_message.yaml
    paramters:
        level: DEBUG

or to create a template resource:

template_resources:
  - label: log_message_foo
    source: https://raw.githubusercontent.com/benthosdev/benthos/main/config/template_examples/processor_log_and_drop.yaml
    paramters:
        level: DEBUG
template_resources:
  - label: log_message_bar
    source: ./templates/processor_log_and_drop.yaml # could be a remote address as well
    paramters:
        level: DEBUG
Jeffail commented 1 year ago

hey @natenho, I'll mark this as a cool idea, we could potentially add remote imports to the CLI flag so you could do benthos -t ./something.yaml -t http://example.com/gimitemplates -c ./foo.yaml. Adding it to the config is harder as there'd be a lot of dynamic behaviour added at parse time, which has some significant side effects.