CloudWatch sink group_name can't contain templated variables during validate

archoversight commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

vector 0.19.0 (x86_64-unknown-linux-gnu da60b55 2021-12-28)

Vector Configuration File

[sinks.cloudwatch]
type = "aws_cloudwatch_logs"
inputs = ["*__out"]
create_missing_group = true
create_missing_stream = true
group_name = "{{ file }}"
compression = "none"
region = "us-east-1"
stream_name = "{{ host }}"
encoding.codec = "json"

Debug Output

# VECTOR_CONFIG_DIR=/etc/vector/ns2/,/etc/vector/sinks/ VECTOR_CONFIG=/etc/vector/vector.toml RUST_BACKTRACE=full vector -v validate
√ Loaded ["/etc/vector/vector.toml", "/etc/vector/ns2", "/etc/vector/sinks"]
√ Component configuration
2022-01-11T04:47:16.764139Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=DescribeLogGroups failed: 1 validation error detected: Value '{{ file }}' at 'logGroupNamePrefix' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+ component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch component_name=cloudwatch
x Health check for "cloudwatch" failed

Expected Behavior

vector validate should continue without issues

Actual Behavior

vector validate fails

Example Data

Not Applicable

Additional Context

Not Applicable

References

None

spencergilbert commented 2 years ago

Similar to https://github.com/vectordotdev/vector/issues/10465, feels like we need to add a default in these cases or setup a healthcheck that doesn't require the group_name/topic to be present for the check.

BryanStenson-okta commented 2 years ago

I think it's worth mentioning the repro config file @archoversight uses contains the example, suggested templating details for group_name and stream_name:

https://vector.dev/docs/reference/configuration/sinks/aws_cloudwatch_logs/#group_name (see examples) https://vector.dev/docs/reference/configuration/sinks/aws_cloudwatch_logs/#stream_name (see examples)

BryanStenson-okta commented 2 years ago

fwiw, despite the validation error, i'm still seeing logs shipped to cloudwatch. (image: timberio/vector:0.19.0-distroless-libc)

spencergilbert commented 2 years ago

fwiw, despite the validation error, i'm still seeing logs shipped to cloudwatch. (image: timberio/vector:0.19.0-distroless-libc)

👍 unless you've required healthcecks to pass the failure won't block running (--require-healthy)

archoversight commented 2 years ago

By default the systemd won't let vector continue starting if the prestart command fails, so I had to explicitly remove the pre-start command.

mmerickel commented 1 year ago

Bumping this as I ran into it as well on vector 0.26.0. It definitely is a bad default as group_name is documented to support templates. The healthcheck needs to account for that and do something else.

2022-12-23T00:53:51.225148Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2022-12-23T00:53:51.225310Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2022-12-23T00:53:51.225387Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2022-12-23T00:53:51.227185Z  INFO vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="ip-10-34-6-71.us-east-2.compute.internal"
2022-12-23T00:53:51.239782Z  INFO vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2022-12-23T00:53:51.307101Z  INFO vector::topology::running: Running healthchecks.
2022-12-23T00:53:51.311137Z  INFO vector: Vector has started. debug="false" version="0.26.0" arch="x86_64" revision="c6b5bc2 2022-12-05"
2022-12-23T00:53:51.311388Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=off
2022-12-23T00:53:51.319409Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs component_name=kubernetes_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2022-12-23T00:53:51.377168Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=DescribeLogGroups failed: InvalidParameterException: 1 validation error detected: Value '/aws/eks/main/vector/test/{{ kubernetes.pod_namespace
 }}' at 'logGroupNamePrefix' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+ component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch component_name=cloudwatch

vectordotdev / vector