ec2:tag substitutions not working in a spark context over ec2

[ebarault]

Hi @kdgregory,

Thank you so much for the work in this series of appenders !

I'm using the log4j 1.x appender (log4j1-aws-appenders:2.4.1 + aws-shared:2.4.1) with spark clusters running in docker on AWS EC2 instances. I've installed the ec2 client: com.amazonaws:aws-java-sdk-ec2:1.12.192

Here is an excerpt of my config and next the only error log I get.

log4j.appender.cloudwatch.logGroup=/cluster/{ec2:tag:aws_env}

log4j:ERROR CloudWatchAppender(cloudwatch): invalid log group name: /cluster/{ec2:tag:aws_env}

Other substitutions work, such as {date}

Can you think of something ?

Note: I also tested with an ssm substitution {ssm:/my/key} and it also fails with a similar error

[kdgregory]

@ebarault - I won't have time to look into this until the weekend, but my gut feeling is that it's the Docker network config.

The ec2:tag substitutions need the current instance ID, which the SDK retrieves from a link-local IP address. If you're using the default "bridged" Docker network, I don't think that address will be available. Instead, try invoking Docker with --network host (there are some other issues with that, but it seems that you're not yet in production).

The ssm subsitution is more puzzling: if you're able to access the Internet at all you should be able to perform that substitution.

I'm 90% sure that the lookup code will report any errors to the Log4J internal logger, so setting the "debug" system property as described in the troubleshooting guide may give you more information. But based on your question you might already have done this, and it's a year since I've been in that code, so I may be imagining that it does that.

[kdgregory]

config.zip @ebarault

After looking into this, I was wrong about the EC2 instance metadata not being available in Docker -- my test program (POM, config, and Dockerfile attached) worked without a problem -- note that it doesn't quite do the same thing as you're doing, but exercises the substitutions).

However, I realized that there's a much simpler explanation for your issue: you're using version 2.4.1 of the library, and these substitutions were not available until version 3. The current version is 3.0.1, and note that there's an extra JAR that's required to support the AWS SDK version.

[ebarault]

ok i see @kdgregory. Thank you, i'll try to upgrade to version 3.0.1.