search

aws / aws-cloudtrail-processing-library

The AWS CloudTrail Processing Library helps Java developers to easily consume and process log files from AWS CloudTrail.
Apache License 2.0
87 stars 30 forks source link

Application not polling from SQS Queue #5

Closed mikals0ft closed 7 years ago

mikals0ft commented 8 years ago

Hi,
I tried to run the sample application, SampleApp. However, it constantly returns: pollQueue is false , and latency is 202832 milliseconds. This seems to be from the SampleProgressReporter, and the executor never seems to go past the pollQueue phase. Any ideas on how to fix this? Thanks.

Best, Michael Qiu

simongjh-zz commented 8 years ago

Hi Mike

Thanks for bring this issue up. From the logs, the polling queue operation took longer than 20 seconds, which most likely reached the long pulling time limit. Before we dive deep, I would like to collect a few pieces of information from you:

  1. Is this issue still happening? Did you try to restart your application?
  2. Is this just happened today? Or it worked fine, and just stopped working?
  3. Do you have the queue setup correctly? Can you double check the permission of the queue, make sure the queue is in the right region as you configured and has correct permission to allow your application to pull messages.
  4. Is there any CloudTrail notification messages in the queue? If not, can you wait and generate a few CloudTrail notifications by invoke some AWS APIs?
  5. Lastly, any other error message you saw in your application log?

Appreciate if you could double check above mentioned cases, and with those additional information, we could help to identify the root cause.

Thanks again to use this library and report issue.

mikals0ft commented 8 years ago

Hi Simon Thanks for the reply.

  1. The issue is still happening, even if I restart the application.
  2. I ran it for the first time yesterday and it didn't work. I tried again today, and same result.
  3. I have a queue on SQS set up, and it is in us-east-1, as I have configured in my cloudtrail properties file. I have enabled everybody to perform all SQS actions for this queue.
  4. I have my queue subscribed to an SNS topic, and this one generates messages pretty frequently (once every minute or 2).
  5. This showed up in the logs:

Aug 04, 2016 10:20:16 AM com.amazonaws.http.AmazonHttpClient executeHelper INFO: Unable to execute HTTP request: connect timed out java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:524) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403) at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:118) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2339) at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1072) at com.amazonaws.services.cloudtrail.processinglibrary.manager.SqsManager.pollQueue(SqsManager.java:126) at com.amazonaws.services.cloudtrail.processinglibrary.reader.EventReader.getSources(EventReader.java:114) at com.amazonaws.services.cloudtrail.processinglibrary.AWSCloudTrailProcessingExecutor$ScheduledJob.run(AWSCloudTrailProcessingExecutor.java:172) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Edit: I tried the receive-message command through the aws cli, and everything seems to be working fine, so it looks like my queue is indeed configured correctly and my access keys are valid.

Thanks! Best, Michael

simongjh-zz commented 8 years ago

Based on the error message, seems related to the timeout (socket and connection) setting in the SQS client. Did you pass in your own SQS client or use the default SQS client? If you pass in your own SQS client, can you let me know your timeout setting?

mikals0ft commented 8 years ago

I used the default SQS client.

mingma7 commented 7 years ago

Hi Mike,

I wonder if this issue is still happening. If so, feel free to reopen the issue.

For your reference, I ran the sample application from scratch in the following steps:

  1. Update the awscloudtrailprocessinglibrary.properties with at least following changes:
    1. accessKey
    2. secretKey
    3. sqsUrl
    4. sqsRegion
  2. Could you verify your accessKey and secretKey are matched, active and have rights to access the SQS queue you configured above?
  3. Could you verify your sqs is subscribed to the right SNS Topic? (Easy way to verify: does your queue have messages coming?)

If all of above steps are configured correct, you are expected to see the following message:

INFO: pollQueue is true , and latency is 288 milliseconds. Oct 10, 2017 1:40:30 PM sample.SampleProgressReporter reportEnd INFO: parseMessage is true , and latency is 1 milliseconds.

Please feel free let us know if you have any question.

-Ming