search

sherzberg / graylog-plugin-s3

Apache License 2.0
16 stars 11 forks source link

Plugin fails to load s3 log #11

Closed Tibingeo closed 6 years ago

Tibingeo commented 6 years ago

@sherzberg This is the error I get 

2018-03-06T14:04:00.367Z ERROR [S3Subscriber] Could not read messages from SNS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.
com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/
        at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:115) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1740) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1716) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1380) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1356) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at org.sherzberg.graylog.aws.inputs.s3.notifications.S3SQSClient.getNotifications(S3SQSClient.java:48) ~[graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at org.sherzberg.graylog.aws.inputs.s3.S3Subscriber$Processor.run(S3Subscriber.java:77) [graylog-plugin-s3-2.3.0-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Th

why is this trying to get message from sns as sns already pushed message to sqs, this plugin should access sqs instead of sns

Tibingeo commented 6 years ago

My graylog sever was running on a aws instance, I got this fixed by adding IAM role to my aws instance where graylog was running. Reference https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html