awslabs / kinesis-agent-windows

An extensible Windows agent that ingests logs and metrics to AWS services such as Kinesis Stream, Kinesis Firehose, CloudWatch Logs and CloudWatch.
Apache License 2.0
39 stars 22 forks source link

Amazon.KinesisFirehose.AmazonKinesisFirehoseException: Signature expired #7

Open SabaBaig opened 5 years ago

SabaBaig commented 5 years ago

I've setup a Kinesis agent on EC2 instance having windows operating system. Its getting me the following error for multiple sinks I've specifiedin my appsetting file. This error occurs randomly. There doesn't seem to be any connection lost issue since its running on EC2 instance also, the clock is not slow.

Kindly guide what could be wrong here and what can be done to avoid it?

2019-06-09 18:25:31.1244 Amazon.KinesisTap.Hosting.LogManager ERROR KinesisFirehoseSink client dev_dw_fh_s3_dl_inventtrans_snk exception: Amazon.KinesisFirehose.AmazonKinesisFirehoseException: Signature expired: 20190609T232013Z is now earlier than 20190609T232031Z (20190609T232531Z - 5 min.) ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.HttpRequest.d16.MoveNext() --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpRequest.d16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.Unmarshaller.d3`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorHandler.d51.MoveNext() --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.d9`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CredentialsRetriever.d71.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Amazon.Runtime.Internal.RetryHandler.d10`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.d91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorCallbackHandler.d5`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.MetricsHandler.d1`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.KinesisTap.AWS.KinesisFirehoseSink.d__7.MoveNext()

aspcompiler commented 5 years ago

Hi @SabaBaig, thanks for your feedback. I just need a little more information:

  1. Your machine is an EC2 instance. You get your credential from instance role, correct?
  2. Which region is your EC2 instance in? Is your Firehose in the same region?
  3. You have some data going through, right? Do you have an estimate on number of errors you are getting versus number of recordings going through?
  4. Does this random error occur all the time or only specific time?
singhharvir commented 4 years ago

Hi SabaBaig@, can you please answer to above questions?

skumawat1984 commented 3 years ago

SabaBaig@

A new beta build is available, this build has multiple fixes,. Pls check if your issue is resolved with this build. Details available at https://github.com/awslabs/kinesis-agent-windows/releases/tag/1.2.3.3