aws / amazon-cloudwatch-agent

CloudWatch Agent enables you to collect and export host-level metrics and logs on instances running Linux or Windows server.
MIT License
433 stars 196 forks source link

OnPremise - Incorrect hostname when device behind a wifi network login page. #184

Open farrantch opened 3 years ago

farrantch commented 3 years ago

We have installed the agent on a device located on a wifi network that redirects to a login page before allowing access to the network.

What I believe is happening - First the agent checks to see if it's an EC2 instance by trying to hit the meta-data service, but the local network is redirecting this request to a login page. The entire contents of this page is then saved as the device hostname.

We are using the hostname within our log-group name. So then an error is thrown when trying to create the log group because the name limit is 512 characters and the content of the web page is much longer than that.

Potential solution - Parse the returned page to make sure it is the ec2 meta-data service that is responding.

pingleig commented 3 years ago

😅 The logic is

But I think it won't get triggered if the agent is configured as onPrem using -m onPremise I think you are using default, which is auto detection without validation

func initEC2UtilSingleton() (newInstance *ec2Util) {
    newInstance = &ec2Util{Region: "", PrivateIP: ""}
    if context.CurrentContext().Mode() == config.ModeOnPrem {
        return
    }

    if info, e := md.GetMetadata("hostname"); e == nil {
        newInstance.Hostname = info
    } else {
        log.Println("E! getting hostname from EC2 metadata fail: ", e)
    }
}

https://github.com/aws/amazon-cloudwatch-agent/blob/master/packaging/dependencies/amazon-cloudwatch-agent-ctl#L56-L59

        -m: mode
            ec2:                                    indicate this is on ec2 host.
            onPremise:                              indicate this is on onPremise host.
            auto:                                   use ec2 metadata to determine the environment, may not be accurate if ec2 metadata is not available for some reason on EC2.

There are some inconsistencies in our logic, the credential has fall backed to static creds instead of using ec2 instance profile, while the hostname logic took wifi logic page as valid ec2 metadata api response. I think we can add a logic to avoid case like this, though the best approach for your use case should be specifying onPremise

btw: in aws go sdk, the ec2metadata API just copy the content without decoding

farrantch commented 3 years ago

Ahh, I can definitely move forward with the -m option. Thank you!!

farrantch commented 3 years ago

So turns out I was already using the -m onPremise option :( So I am going to re-open this ticket.

Perhaps potential solution now is to verify -m onPremise uses os.hostname by default?

jhnlsn commented 2 years ago

I'm assuming that the current logic relies on the request to the metadata endpoint to fail as the decision to use alternative means to lookup the hostname. This should be pretty easy to replicate by spoofing the response from the metadata endpoint ip to verify this behavior.

SaxyPandaBear commented 2 years ago

@farrantch Have you tried using the {local_hostname} placeholder instead of {hostname}?

I didn't know about this until just now as I was looking at the code in ./translator/translate/util/placeholderUtil.go

func getHostName() string {
    if hostname, err := os.Hostname(); err == nil {
        return hostname
    } else {
        log.Println("E! getHostName: ", err)
        return unknownHostname
    }
}