Closed rnhurt closed 1 year ago
Possibly related, https://github.com/vectordotdev/vector/discussions/16627
It looks like some sort of credentials issue, so I removed the ~/.aws
directory to see if it was getting confused somewhere. The error is still present and there was no change in behaviour.
@rnhurt could you try v0.27.1 as well? We think this may be a regression in v0.28.0.
Assuming this is not present in 0.27
likely culprit would be the the upgrade to the AWS SDK, there was also an upgrade to 0.53
included in this release, but 0.54
had some specific changes to credentials.
@rnhurt I'd also be curious to know if you get the same InvalidSignatureException
error that @samidalouche saw.
I have encountered the same error (https://github.com/vectordotdev/vector/discussions/16627)
I reverted to this version, and the problem went away.
yum install -y vector-0.27.1-1
@rnhurt I'd also be curious to know if you get the same
InvalidSignatureException
error that @samidalouche saw.
I didn't see the InvalidSignatureException
error but I'm not sure I ran with the correct DEBUG flags. 🤷
Here are the logs from my successful run using v0.27.1:
2023-02-28T15:04:30.278368Z INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-28T15:04:30.278507Z INFO vector::app: Log level is enabled. level="debug"
2023-02-28T15:04:30.278569Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-02-28T15:04:30.279299Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-02-28T15:04:30.279904Z DEBUG vector::topology::builder: Building new source. component=tailscale_logs
2023-02-28T15:04:30.280148Z DEBUG vector::topology::builder: Building new sink. component=cloudwatch_logs
2023-02-28T15:04:30.280891Z DEBUG hyper_rustls::config: with_native_roots processed 124 valid and 0 invalid certs
2023-02-28T15:04:30.280936Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T15:04:30.280979Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T15:04:30.281014Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T15:04:30.281047Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T15:04:30.281070Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T15:04:30.281092Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T15:04:30.281126Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T15:04:30.281148Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T15:04:30.291511Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.300081Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.318032Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.326679Z DEBUG vector_core::tls::settings: Fetching system root certs.
2023-02-28T15:04:30.336450Z INFO vector::topology::running: Running healthchecks.
2023-02-28T15:04:30.336490Z DEBUG vector::topology::running: Connecting changed/added component(s).
2023-02-28T15:04:30.336518Z DEBUG vector::topology::running: Configuring outputs for source. component=tailscale_logs
2023-02-28T15:04:30.336540Z DEBUG vector::topology::running: Configuring output for component. component=tailscale_logs output_id=None
2023-02-28T15:04:30.336558Z DEBUG vector::topology::running: Connecting inputs for sink. component=cloudwatch_logs
2023-02-28T15:04:30.336575Z DEBUG vector::topology::running: Adding component input to fanout. component=cloudwatch_logs fanout_id=tailscale_logs
2023-02-28T15:04:30.336620Z DEBUG vector::topology::running: Spawning new source. key=tailscale_logs
2023-02-28T15:04:30.336653Z DEBUG vector::topology::running: Registered new allocation group. component_kind="source" component_type="journald" component_id="tailscale_logs" group_id="2"
2023-02-28T15:04:30.336692Z DEBUG vector::topology::running: Registered new allocation group. component_kind="sink" component_type="aws_cloudwatch_logs" component_id="cloudwatch_logs" group_id="3"
2023-02-28T15:04:30.336750Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor starting.
2023-02-28T15:04:30.336765Z INFO vector: Vector has started. debug="false" version="0.27.1" arch="aarch64" revision="19a51f2 2023-02-22"
2023-02-28T15:04:30.336798Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump starting.
2023-02-28T15:04:30.336827Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-28T15:04:30.337059Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source starting.
2023-02-28T15:04:30.337137Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink starting.
2023-02-28T15:04:30.336998Z DEBUG aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:30.337310Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=environment variable not set
2023-02-28T15:04:30.337358Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2023-02-28T15:04:30.337383Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2023-02-28T15:04:30.337402Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2023-02-28T15:04:30.337436Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=29
2023-02-28T15:04:30.337458Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2023-02-28T15:04:30.337475Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2023-02-28T15:04:30.337498Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2023-02-28T15:04:30.337516Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2023-02-28T15:04:30.337551Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=profile `default` did not contain credential information
2023-02-28T15:04:30.337571Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
2023-02-28T15:04:30.337599Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
2023-02-28T15:04:30.337617Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-02-28T15:04:30.337747Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:send_operation{operation="get-token" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.337926Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.09498622198941609
2023-02-28T15:04:30.337949Z INFO source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Starting journalctl.
2023-02-28T15:04:30.338027Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:send_operation{operation="get-token" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.338191Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2023-02-28T15:04:30.338679Z DEBUG hyper::proto::h1::io: parsed 6 headers
2023-02-28T15:04:30.338694Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2023-02-28T15:04:30.338716Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.338873Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.339019Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.339074Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2023-02-28T15:04:30.339379Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T15:04:30.339390Z DEBUG hyper::proto::h1::conn: incoming body is content-length (49 bytes)
2023-02-28T15:04:30.339396Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.339452Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=tailscale-account-iamrTailscaleRole-1UHIDHJGVOPRZ
2023-02-28T15:04:30.339497Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connecting to 169.254.169.254:80
2023-02-28T15:04:30.339602Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}: hyper::client::connect::http: connected to 169.254.169.254:80
2023-02-28T15:04:30.339661Z DEBUG hyper::proto::h1::io: flushed 356 bytes
2023-02-28T15:04:30.340003Z DEBUG hyper::proto::h1::io: parsed 8 headers
2023-02-28T15:04:30.340014Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1582 bytes)
2023-02-28T15:04:30.340025Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.340105Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2023-02-28T15:04:30.340323Z DEBUG hyper::client::connect::dns: resolving host="logs.us-east-1.amazonaws.com"
2023-02-28T15:04:30.341742Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::connect::http: connecting to 44.202.79.205:443
2023-02-28T15:04:30.344505Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::connect::http: connected to 44.202.79.205:443
2023-02-28T15:04:30.347936Z DEBUG hyper::proto::h1::io: flushed 2032 bytes
2023-02-28T15:04:30.372791Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:30.373075Z DEBUG hyper::proto::h1::conn: incoming body is content-length (240 bytes)
2023-02-28T15:04:30.373225Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:30.373339Z DEBUG send_operation{operation="DescribeLogGroups" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:30.373707Z INFO vector::topology::builder: Healthcheck passed.
2023-02-28T15:04:31.414132Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:31.414250Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:31.414275Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=100
2023-02-28T15:04:31.414349Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.414389Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.414464Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.414549Z DEBUG hyper::proto::h1::io: flushed 2084 bytes
2023-02-28T15:04:31.428363Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.428392Z DEBUG hyper::proto::h1::conn: incoming body is content-length (17 bytes)
2023-02-28T15:04:31.428550Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.428598Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.428676Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Provided stream does not exist; creating a new one.
2023-02-28T15:04:31.428758Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.428811Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.428912Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.429012Z DEBUG hyper::proto::h1::io: flushed 2065 bytes
2023-02-28T15:04:31.471472Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.471502Z DEBUG hyper::proto::h1::conn: incoming body is empty
2023-02-28T15:04:31.471690Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="CreateLogStream" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.471752Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Stream created. name=f431275382ea4a47885bce73bceb6781
2023-02-28T15:04:31.471810Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.471854Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.471944Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.472043Z DEBUG hyper::proto::h1::io: flushed 2084 bytes
2023-02-28T15:04:31.485395Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.485424Z DEBUG hyper::proto::h1::conn: incoming body is content-length (226 bytes)
2023-02-28T15:04:31.485480Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.485534Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="DescribeLogStreams" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.485644Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Stream found. stream=Some("f431275382ea4a47885bce73bceb6781")
2023-02-28T15:04:31.485663Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs. token=None
2023-02-28T15:04:31.486201Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:31.486257Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:31.486445Z DEBUG hyper::client::connect::dns: resolving host="logs.us-east-1.amazonaws.com"
2023-02-28T15:04:31.486797Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::connect::http: connecting to 44.202.79.192:443
2023-02-28T15:04:31.489894Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::connect::http: connected to 44.202.79.192:443
2023-02-28T15:04:31.494727Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494789Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494845Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494880Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494917Z DEBUG hyper::proto::h1::io: flushed 16384 bytes
2023-02-28T15:04:31.494949Z DEBUG hyper::proto::h1::io: flushed 13718 bytes
2023-02-28T15:04:31.554980Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:31.555013Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:31.555230Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:31.555279Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:31.555372Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49637405755031694129084131660689759327850337978875380530")
2023-02-28T15:04:35.338094Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.009927846799178427
2023-02-28T15:04:36.088027Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: tower::buffer::worker: service.ready=true processing request2023-02-28T15:04:36.088132Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:36.088157Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=2
2023-02-28T15:04:36.088265Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "signingRegion": String("us-east-1")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:36.088336Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:36.088436Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:36.088548Z DEBUG hyper::proto::h1::io: flushed 4061 bytes
2023-02-28T15:04:36.105390Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:36.105418Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:36.105450Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:36.105501Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:36.105575Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49628116737211695456459775456880034164813271731714131042")
2023-02-28T15:04:40.338219Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000992803759438506
2023-02-28T15:04:42.057163Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:42.057247Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: tower::buffer::worker: service.ready=true processing request
2023-02-28T15:04:42.057271Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=1
2023-02-28T15:04:42.057389Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: aws_endpoint: resolved endpoint endpoint=Endpoint { url: "https://logs.us-east-1.amazonaws.com/", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("us-east-1"), "name": String("sigv4")})])} } base_region=Some(SigningRegion("us-east-1"))
2023-02-28T15:04:42.057441Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2023-02-28T15:04:42.057535Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: reuse idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:42.057673Z DEBUG hyper::proto::h1::io: flushed 3110 bytes
2023-02-28T15:04:42.078286Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-02-28T15:04:42.078320Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-28T15:04:42.078364Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-02-28T15:04:42.078420Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}:send_operation{operation="PutLogEvents" service="cloudwatchlogs"}: hyper::client::pool: pooling idle connection for ("https", logs.us-east-1.amazonaws.com)
2023-02-28T15:04:42.078488Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49636311869716377971442486284532261404947306999409803410")
2023-02-28T15:04:45.338839Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.00010497450932245926
2023-02-28T15:04:50.338313Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::utilization: utilization=0.000010566938244949766
^C2023-02-28T15:04:53.560864Z INFO vector: Vector has stopped.
2023-02-28T15:04:53.561028Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source finished normally.
2023-02-28T15:04:53.561054Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump finished normally.
2023-02-28T15:04:53.561104Z DEBUG source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::topology::builder: Source pump supervisor task finished normally.
2023-02-28T15:04:53.561139Z DEBUG sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}: vector::topology::builder: Sink finished normally.
I didn't see the
InvalidSignatureException
error but I'm not sure I ran with the correct DEBUG flags. 🤷
For some reason, setting the RUST_LOG=aws_smithy_http=trace
env variable in /etc/default/vector
did not produce the extra logs, so I had to manually run the vector command-line outside of systemd to get them
I didn't see the
InvalidSignatureException
error but I'm not sure I ran with the correct DEBUG flags. 🤷For some reason, setting the
RUST_LOG=aws_smithy_http=trace
env variable in/etc/default/vector
did not produce the extra logs, so I had to manually run the vector command-line outside of systemd to get them
I did run it outside of systemd but still didn't see anything new. 🤔
Thanks for checking! I'll be looking at this today, appreciate the thorough reports.
Well, I apparently didn't run it with the correct flags or something. Running RUST_LOG=aws_smithy_http=trace vector
gives me this, which does include the InvalidSignatureException
error that @samidalouche is seeing on v0.28.
[root@ip-10-202-11-141 ~]# RUST_LOG=aws_smithy_http=trace vector
2023-02-28T15:12:13.090276Z INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-28T15:12:13.093994Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-02-28T15:12:13.098707Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-02-28T15:12:13.244594Z INFO vector::topology::running: Running healthchecks.
2023-02-28T15:12:13.247553Z INFO vector: Vector has started. debug="false" version="0.28.0" arch="aarch64" revision="971c594 2023-02-27"
2023-02-28T15:12:13.247588Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-02-28T15:12:13.252260Z INFO source{component_kind="source" component_id=tailscale_logs component_type=journald component_name=tailscale_logs}: vector::sources::journald: Starting journalctl.
2023-02-28T15:12:13.296709Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: service error component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch_logs component_name=cloudwatch_logs
2023-02-28T15:12:14.281053Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=1
2023-02-28T15:12:14.292588Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:12:14.292633Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:12:14.295423Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("fd193320-e53f-4f07-bac5-977e9a32d55b"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("fd193320-e53f-4f07-bac5-977e9a32d55b"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "fd193320-e53f-4f07-bac5-977e9a32d55b", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:12:13 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-02-28T15:12:14.295490Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-02-28T15:13:08.329746Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=6
2023-02-28T15:13:08.342241Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been rate limited 1 times.
2023-02-28T15:13:08.342268Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:13:08.342292Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:13:08.342357Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("c21db376-0074-4e15-9e13-3f06b4fb74f0"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("c21db376-0074-4e15-9e13-3f06b4fb74f0"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "c21db376-0074-4e15-9e13-3f06b4fb74f0", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:13:07 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=2 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-02-28T15:13:08.342412Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=6 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-02-28T15:13:21.300819Z INFO sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=3
2023-02-28T15:13:21.312694Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been rate limited 1 times.
2023-02-28T15:13:21.312723Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: service error internal_log_rate_limit=true
2023-02-28T15:13:21.312744Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2023-02-28T15:13:21.312808Z ERROR sink{component_kind="sink" component_id=cloudwatch_logs component_type=aws_cloudwatch_logs component_name=cloudwatch_logs}:request{request_id=3}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ServiceError(ServiceError { source: DescribeLogStreamsError { kind: Unhandled(Unhandled { source: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("180c4f5e-16f4-4b07-a398-71c3fdc9418b"), extras: {} } }), meta: Error { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), request_id: Some("180c4f5e-16f4-4b07-a398-71c3fdc9418b"), extras: {} } }, raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "180c4f5e-16f4-4b07-a398-71c3fdc9418b", "content-type": "application/x-amz-json-1.1", "content-length": "229", "date": "Tue, 28 Feb 2023 15:13:20 GMT", "connection": "close"}, body: SdkBody { inner: Once(Some(b"{\"__type\":\"InvalidSignatureException\",\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }))) request_id=3 error_type="request_failed" stage="sending" internal_log_rate_limit=true
...
Update! It was actually the upgrade to 0.53
, hoping to have a fix by the end of the day.
We also saw this on a different service:
2023-02-28T05:00:36.992307Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeDeliveryStream failed: service error component_kind="sink" component_type="aws_kinesis_firehose" component_id=firehose_logs component_name=firehose_logs
I can open a separate ticket if you like, but it seems very similar to this.
We also saw this on a different service:
2023-02-28T05:00:36.992307Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeDeliveryStream failed: service error component_kind="sink" component_type="aws_kinesis_firehose" component_id=firehose_logs component_name=firehose_logs
I can open a separate ticket if you like, but it seems very similar to this.
We were able to track the issue down. Unfortunately it affects all AWS components. We aim to fix it and release it as part of 0.28.1 shortly. For now, I'd recommend using 0.27.1.
Hi I am still facing same problem. just want to check is this been fixed? here is my configurations
[sources.http]
type = "http_server"
address = "0.0.0.0:8080"
encoding = "json"
healthcheck = true
[sinks.aws_cloudwatch_logs]
type = "aws_cloudwatch_logs"
inputs = ["http"]
compression = "none"
create_missing_group = true
create_missing_stream = true
group_name = "test_opentelemetry"
region = "ap-south-1"
stream_name = "test"
encoding.codec = "json"
here is the error message
nuc2kor@PUA-V-0001T:~$ sudo vector -c /etc/vector/vector.toml
2023-11-01T06:49:53.517964Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-11-01T06:49:53.520824Z INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-11-01T06:49:53.624274Z INFO vector::topology::running: Running healthchecks.
2023-11-01T06:49:53.624401Z INFO vector: Vector has started. debug="false" version="0.33.0" arch="x86_64" revision="89605fb 2023-09-27 14:18:24.180809939"
2023-11-01T06:49:53.624415Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-11-01T06:49:53.624612Z INFO source{component_kind="source" component_id=http component_type=http_server component_name=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:8080
2023-11-01T06:49:54.641029Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: failed to construct request component_kind="sink" component_type="aws_cloudwatch_logs" component_id=aws_cloudwatch_logs component_name=aws_cloudwatch_logs
2023-11-01T06:50:08.535133Z INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=2
@Strange21 this issue was fixed in v0.28.1. I'm running the latest version and haven't had any problems since it was fixed. What version of Vector are you running? Also, check to make sure the IAM permissions are correct and include the "DescribeLogGroups" permission.
A note for the community
Problem
It looks like the new release doesn't like my CloudWatch logs sink configuration. I'm running Vector on AWS Amazon Linux 2 to push Tailscale logs to CloudWatch logs. This configuration has worked perfectly for months and this morning it broke when the system updated to the latest version of Vector.
I've confirmed that I can access the CloudWatch logs from that machine using the AWS CLI, so I don't think it's a permissions problem. This instance has an IAM Profile assigned to it and is part of a autoscaling group that refreshes daily.
Configuration
Example Data
No response
Additional Context
No response
References
No response