Configuration for use with AWS Lambda

[efenderbosch]

AWS Lambda supports using Log4J to log to CloudWatch.

We are currently evaluating CloudWatch vs Sentry and it seems like they will both be useful for different scenarios.

However, we are getting errors like below when the Lambda is "thawed" after an extended period of time and reused.

SLF4J: A number (12) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
2017-03-20 14:11:17.593 INFO com.getsentry.raven.DefaultRavenFactory - Using an HTTP connection to Sentry.
2017-03-20 14:11:17.736 INFO tsentry.raven.connection.AsyncConnection - Gracefully shutdown sentry threads.
2017-03-20 14:11:17.737 INFO tsentry.raven.connection.AsyncConnection - Shutdown finished.
log4j:ERROR Attempted to append to closed appender named [Sentry].
log4j:ERROR Attempted to append to closed appender named [Sentry].
2017-03-20 14:11:17.742 INFO com.getsentry.raven.DefaultRavenFactory - Using an HTTP connection to Sentry.

Our log4j.properties is pretty simple:

log4j.appender.LAMBDA=com.amazonaws.services.lambda.runtime.log4j.LambdaAppender
log4j.appender.LAMBDA.layout=org.apache.log4j.PatternLayout
log4j.appender.LAMBDA.layout.conversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{AWSRegion}] [%X{AWSInitialRequestId}] [%X{AWSRequestId}] [%X{AWSFunctionName}] [%X{AWSFunctionVersion}] %-5p %-40.40c - %m%n

log4j.rootLogger=INFO, LAMBDA, Sentry

log4j.logger.com.company=DEBUG, LAMBDA, Sentry
log4j.additivity.com.company=false

log4j.appender.Sentry=com.getsentry.raven.log4j.SentryAppender
log4j.appender.Sentry.threshold=WARN

# Set the Sentry DSN
log4j.appender.Sentry.dsn=<redacted>

Is there anything we can or should do during initialization to ensure that Sentry's http connections are still valid, or is there a way to assume they are invalid and force initialization. Maybe a custom RavenFactory?

[bretthoerner]

Interesting, it seems like Lambda closes the Appenders when it freezes the app?

I'll see what I can find out, in the meantime you might have better luck with the non-logging usage of Raven: https://docs.sentry.io/clients/java/usage/

With that you could wrap the main entry point in a try/catch and send all exceptions to Sentry that way.

[bretthoerner]

It looks like the lambda logger noops for close: https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-log4j/src/main/java/com/amazonaws/services/lambda/runtime/log4j/LambdaAppender.java#L47-L48

And redirects logs to stdout: https://github.com/aws/aws-lambda-java-libs/blob/60f9750ccc3f799d9b773e0178f1da8e52773990/aws-lambda-java-core/src/main/java/com/amazonaws/services/lambda/runtime/LambdaRuntime.java#L8-L12

So I guess for Lambda we could try just making our SentryAppender close noop? But I think using the "manual Raven usage" linked above will work best if you want to get it working now.

[efenderbosch]

Ok, thanks. I'm not sure we want to update all our code to use the manual usage. We might just delay Sentry integration until this is "fixed."

[bretthoerner]

@efenderbosch is there any chance you can try this? You should be able to have this class in your own codebase:

package com.getsentry.raven.log4j;

public class LambdaSentryAppender extends SentryAppender {
    @Override
    public void close() {

    }
}

And then configure log4j to use the appender com.getsentry.raven.log4j.LambdaSentryAppender

[efenderbosch]

I've added LambdaSentryAppender, built and deployed one of our Lambda functions. I'll let it freeze, then test to see what it does during thaw.

[efenderbosch]

That seems to have at least partially fixed it:

SLF4J: A number (12) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
2017-03-22 20:49:56.656 INFO com.getsentry.raven.DefaultRavenFactory - Using an HTTP connection to Sentry.
2017-03-22 20:49:56.793 INFO com.getsentry.raven.DefaultRavenFactory - Using an HTTP connection to Sentry.

It still seems like startup after a thaw is slower than w/o Sentry. I can double check that tomorrow.

[bretthoerner]

Those are just INFO level logs and not errors, so hopefully everything is OK now. If you cause an exception after thaw does it actually successfully send to Sentry?

[bretthoerner]

@efenderbosch Has that solution worked for you? If so I can try to codify it in the SDK.

[efenderbosch]

We're trying to get a product out the door this week, so this has fallen down my priority list. I'll try to test ASAP.

[efenderbosch]

We've been in a dark-production for a week or so now and we're getting alerts from our lambda functions. This seems to be working correctly.

[bretthoerner]

Great, thanks for the update. I'll add the class and document this soon.

[musketyr]

I was able to integrate AWS Lambda with Sentry using io.sentry:sentry-log4j2 with setting async=false inside sentry.properties

[djselzlein]

@musketyr could you provide an example on how you made it work? I have configured SentryAppender in log4j.properties but logs do not get sent to Sentry

[musketyr]

sure. create file sentry.properties next to log4j.properties and add following line inside:

async=false

[djselzlein]

thank you for your help. I have it like you say, but still logs are sent to Sentry only when I use Sentry.capture(ex) but not when I do log.error(ex)

[musketyr]

maybe a difference in configuration - mine looks pretty similar to this one

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.apache.logging.log4j.core,io.sentry.log4j2,com.amazonaws.services.lambda.runtime.log4j2">
    <Appenders>
        <Lambda name="Lambda">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n"/>
        </Lambda>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
        <Sentry name="Sentry"/>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Lambda"/>
            <!-- Note that the Sentry logging threshold is overridden to the WARN level -->
            <AppenderRef ref="Sentry" level="warn"/>
        </Root>
    </Loggers>
</Configuration>

[djselzlein]

I am actually using Lo4j, not Log4j2. So I have my appenders configured on lo4j.properties similar to this:

log = .
log4j.rootLogger = TRACE, Sentry

# Configure the Sentry appender, overriding the logging threshold to the WARN level
log4j.appender.Sentry=io.sentry.log4j.SentryAppender
log4j.appender.Sentry.threshold=WARN

I will try with Log4j2 now, thanks @musketyr

[musketyr]

Ok, Log4J2 is the officially supported logging framework

[Mayank-partsavatar]

@djselzlein did you try log4j2 can you please help me i am unable to get the alerts in sentry

[djselzlein]

@Mayank-partsavatar I ended up using Sentry client library instead of the appender strategy. Did you get it to work?

[adinauer]

Closing this as no recent activity and we have https://github.com/getsentry/sentry-java/issues/1927 . Please feel free to re-open.

To make Sentry work on AWS lambdas, you need to use Sentry.flush() to wait until all the events have been sent out.

[JankoLancer]

Hey, I have flush but I don't get automatic log breadcrumbs to Sentry exceptions. I get the one that I add manually with .addBreadcrumbs but it would be great to get all logs also. Did I make some mistake anywhere or that doesn't work?