search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.5k stars 1.53k forks source link

Inputs with wildcards that match nothing became illegal with 0.17 #9602

Closed flyingflo closed 1 year ago

flyingflo commented 2 years ago

Vector Version

vector 0.17.0 (x86_64-unknown-linux-gnu 3d34cde 2021-10-08)

Vector Configuration File

transforms:
  parse_mylog:
    type: remap
    inputs:
      - source1
      - sources_type_*
      - another_*
      - a_third_*
   [...]

Expected Behavior

We use modular config files with sources for several applications, that are not present on all hosts. We connect them to a common transform with a wildcard input list. With 0.16, this was ok, as long as there was at least one input.

Actual Behavior

Since 0.17, any wildcard input that does not match anything fails the validation.

x Input "sources_type_*" for transform "parse_mylog" doesn't match any components.

This could be intended behaviour, but it comes by surprise and breaks configs.

jszwedko commented 2 years ago

Hi @flyingflo !

Thanks for filing this. This was an intentional change, but we can see how this would break certain use-cases so we are thinking about how to re-enable the previous behavior. Our current thoughts are to introduce a strict flag that can be disabled to turn this error into a warning.

flyingflo commented 2 years ago

Ok, cool. I've already worked around that in my config. I found that there is a flag for vector validate that makes warnings fatal: --deny-warnings. That sounds similar to the strict flag you suggested.

jszwedko commented 1 year ago

Closing in-favor of https://github.com/vectordotdev/vector/issues/13056

bossm8 commented 2 months ago

@flyingflo did you manage to start vector anyway or did you adjust your configs that at least one input matches? Could you share an example? We have a pretty similar use-case where we want to enable opt-in sinks with the config-dir and watch-config options. But we are not able to start vector because it fails hard when there is not at least one matching input.

flyingflo commented 2 months ago

We make sure at least one input matches in all cases. When I needed a dummy source, I created one that reads from /dev/null.

bossm8 commented 2 months ago

We make sure at least one input matches in all cases. When I needed a dummy source, I created one that reads from /dev/null.

Thx! We ended up doing the same but used the demo logs with count set to zero 👍