search

Graylog2 / graylog-s3-lambda

An AWS Lambda function that reads logs from S3 and sends them to Graylog
GNU General Public License v3.0
12 stars 6 forks source link

Update to support Java 11 #20

Open ghost opened 3 years ago

ghost commented 3 years ago

Given that Graylog v3+ supports Java OpenJDK 11 (Amazon Corretto 11), the lambda function should be updated to support Java 11 (Corretto) in the Amazon Lambda function.

ghost commented 3 years ago

I wanted to update this because of the announcement from Amazon regarding Java 8. https://aws.amazon.com/blogs/compute/announcing-migration-of-the-java-8-runtime-in-aws-lambda-to-amazon-corretto/

In testing, using Java 8 (Corretto) broke the function and prevented the logs from being ingested into Graylog. Should we expect this lambda function to continue to work after July 12, 2021?

danotorrey commented 3 years ago

@justinjett Thanks for the follow-up. I attempted to reproduce the issue running the graylog-s3-lambda function on the java8.al2/amazon-corretto-8 runtime, but was unable to. The function ran successfully in my test (and the messages were delivered to Graylog).

Can you please tell me more about the failure you are seeing? Does the function not execute at all, or is an error presented in the log output? If so, can you provide the error and any configuration specifics that will help me reproduce the issue?

ghost commented 3 years ago

@danotorrey I think I've figured out the issue. It appears to be a memory-related issue.

Here are some DEBUG logs from the Lambda function.

START RequestId: 241aa0fc-1c56-431c-9807-8b85c9815ba6 Version: $LATEST
--
2021-07-07 21:05:18 241aa0fc-1c56-431c-9807-8b85c9815ba6 INFO  GraylogS3Function:87 - Log level is now set to [DEBUG].
org.graylog.integrations.s3.codec.CodecProcessor
org.graylog.integrations.s3.CompressionType
com.github.joschi.jadconfig.JadConfig
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
2021-07-07 21:05:18 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:59 - Configuration{, graylogHost='172.22.1.40', graylogPort=12201, connectTimeout=60000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=512, maxInflightSends=512, contentType='application/x.cloudflare.log', compressionType='gzip', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, EdgeResponseStatus, ClientIP, EdgeResponseBytes, CacheCacheStatus, EdgeColoCode, RayID]'}
2021-07-07 21:05:27 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:107 - Graylog host: 172.22.1.40:12201
2021-07-07 21:05:27 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:109 - Attempting to read object [com.amazonaws.services.s3.event.S3EventNotification$S3Entity@6a472554] from S3.
2021-07-07 21:05:33 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:112 - Object read from S3 successfully.
2021-07-07 21:05:49 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [100] messages.
2021-07-07 21:06:11 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [200] messages.
2021-07-07 21:06:25 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [300] messages.
2021-07-07 21:06:40 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [400] messages.
Exception in thread "GelfSenderThread-11" java.lang.OutOfMemoryError: Metaspace
2021-07-07 21:06:54 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [500] messages.
Exception in thread "gelfTcpTransport-1-1" java.lang.OutOfMemoryError: Metaspace
2021-07-07 21:07:09 241aa0fc-1c56-431c-9807-8b85c9815ba6 DEBUG GraylogS3Function:187 - Sent [600] messages.
END RequestId: 241aa0fc-1c56-431c-9807-8b85c9815ba6
REPORT RequestId: 241aa0fc-1c56-431c-9807-8b85c9815ba6  Duration: 120084.91 ms  Billed Duration: 120000 ms  Memory Size: 256 MB Max Memory Used: 178 MB Init Duration: 1690.35 ms
2021-07-07T21:07:17.291Z 241aa0fc-1c56-431c-9807-8b85c9815ba6 Task timed out after 120.08 seconds

When I set the Memory in the function to 512MB, the issue seems to resolve itself. What is the expected configuration for the function (I don't see it listed in the README)? Why does switching from java8 to java8.al2 cause the function to use more resources?

danotorrey commented 3 years ago

Thanks for the update @justinjett

I have opened a new specific issue for the increased memory utilization: https://github.com/Graylog2/graylog-s3-lambda/issues/23#issuecomment-875951664