Container is stopped. On-premise setup.

[mishelen]

Describe the bug I need CWagent instance to intercept logging via statsD. But I'm struggle just to run it in the container.

Steps to reproduce So I am just reusing image from this repository with few updates

FROM ubuntu:latest as build

RUN apt-get update &&  \
    apt-get install -y ca-certificates curl && \
    rm -rf /var/lib/apt/lists/*

RUN curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/${TARGETARCH:-$(dpkg --print-architecture)}/latest/amazon-cloudwatch-agent.deb && \
    dpkg -i -E amazon-cloudwatch-agent.deb && \
    rm -rf /tmp/* && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/config-downloader

FROM scratch

COPY --from=build /tmp /tmp
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /opt/aws/amazon-cloudwatch-agent /opt/aws/amazon-cloudwatch-agent

COPY cloudwatch/common-config.toml /opt/aws/amazon-cloudwatch-agent/etc/
COPY cloudwatch/config.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

# ENV RUN_IN_CONTAINER="True"
# ENV CONFIG_DIR_IN_CONTAINER="/opt/aws/amazon-cloudwatch-agent/etc"
ENTRYPOINT ["/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent"]

Provided configs:

common-config.toml

 [credentials]
    shared_credential_profile = "default"
    shared_credential_file= ".aws/credentials"

amazon-cloudwatch-agent.json

{
  "agent": {
    "run_as_user": "cwagent",
    "region": "us-east-1",
    "debug": true
  },
  "metrics": {
    "namespace": "k6",
    "metrics_collected": {
      "statsd": {
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      }
    }
  }
}

What did you expect to see? I expect to know what happened with launch I expect to set debug and see that in details.

What did you see instead?

1. Issue

D! [EC2] Found active network interface
I! imds retry client will retry 1 timesD! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! could not get hostname without imds v1 fallback enable thus enable fallback
E! [EC2] Fetch hostname from EC2 metadata fail: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/hostname": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: no route to hostD! could not get instance document without imds v1 fallback enable thus enable fallback
E! [EC2] Fetch identity document from EC2 metadata fail: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: connect: no route to host
I! Detected the instance is OnPremise
2024/04/25 09:55:36 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2024/04/25 09:55:36 I! Valid Json input schema.
2024/04/25 09:55:36 D! pipeline hostDeltaMetrics has no receivers
2024/04/25 09:55:36 Configuration validation first phase succeeded
I! Detecting run_as_user...

And then it stops silently and immediately without errors. Setting debug won't change output. Setting run_as_user to cwagent or root or removing it, won't change output. From "secret" conversation there I know that common-config.toml isn't supported. What's the problem? Plus I wasn't able to find how to provide settings from .toml to cwagent.

2. Issue

You probable noticed commented

# ENV RUN_IN_CONTAINER="True"
# ENV CONFIG_DIR_IN_CONTAINER="/opt/aws/amazon-cloudwatch-agent/etc"

Even I need to use them, but then I got even more problems how to setup that:

I! Detected the instance is OnPremise
2024-04-25T10:15:54.649186264Z 2024/04/25 10:15:54 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
2024-04-25T10:15:54.649189074Z /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2024-04-25T10:15:54.649191450Z Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directory 
2024-04-25T10:15:54.649193834Z 2024/04/25 10:15:54 unable to scan config dir /etc/cwagentconfig with error: lstat /etc/cwagentconfig: no such file or directory
2024-04-25T10:15:54.651426573Z 2024/04/25 10:15:54 I! Return exit error: exit code=99
2024-04-25T10:15:54.651437459Z 2024/04/25 10:15:54 I! No json config files found, please provide config, exit now

I want know clear way how to setup container. Which directories and files should be used, and why it ignores default directories.

What version did you use? latest

What config did you use? see above

Environment ubuntu:latest

Additional context Add any other context about the problem here.

[mishelen]

this seems working:

FROM ubuntu:latest as build

RUN apt-get update &&  \
    apt-get install -y ca-certificates curl && \
    rm -rf /var/lib/apt/lists/*

RUN curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/${TARGETARCH:-$(dpkg --print-architecture)}/latest/amazon-cloudwatch-agent.deb && \
    dpkg -i -E amazon-cloudwatch-agent.deb && \
    rm -rf /tmp/* && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl && \
    rm -rf /opt/aws/amazon-cloudwatch-agent/bin/config-downloader

FROM scratch

COPY --from=build /tmp /tmp
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /opt/aws/amazon-cloudwatch-agent /opt/aws/amazon-cloudwatch-agent

COPY cloudwatch/common-config.toml /etc/cwagentconfig/
COPY cloudwatch/config.json /etc/cwagentconfig/amazon-cloudwatch-agent.json

ENV RUN_IN_CONTAINER="True"
ENTRYPOINT ["/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent"]

but now I have other issue, even .toml in so called /etc/cwagentconfig/ it is fail with error, looks like this folder is for jsons.

How to provide content of .toml as environment variables?

[github-actions[bot]]

This issue was marked stale due to lack of activity.

[github-actions[bot]]

Closing this because it has stalled. Feel free to reopen if this issue is still relevant, or to ping the collaborator who labeled it stalled if you have any questions.