InvalidSequenceTokenException The given sequenceToken is invalid. The next expected sequenceToken

Description

Lots of log events generate aws-logger-erros.txt file with "Amazon.CloudWatchLogs.Model.InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is:" errors

Logs

Log Entry :
12/7/2020 3:47:16 PM
  :
  :Amazon.CloudWatchLogs.Model.InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49613091897100174663800395107016604379691865278635293858
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
-------------------------------
Log Entry :
12/7/2020 3:47:16 PM
  :
  :Amazon.CloudWatchLogs.Model.InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49613091897100174663800395107016604379691865278635293858
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at AWS.Logger.Core.AWSLoggerCore.SendMessages(CancellationToken token)
-------------------------------

Environment

Issue found in HostedServices

Build Version: AWS.Logger.Log4net 2.0.0
OS Info: AmazonLinux
Build Environment: Visual Studio + Terminal dotnet
Targeted .NET Platform: netcore 3.1

This is a :bug: bug-report

Hi @ValeryYegorov,

Good morning.

Please share the code solution the might be causing this error. Is it the Lambda code, Serverless application, or may be invalid logger configuration? Was the error happening earlier or started happening recently? We need some information (preferably in the below format), in order to troubleshoot the issue.

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

AWSSDK.Core version used:
Service assembly and version used:
Operating System and version:
Visual Studio version:
Targeted .NET platform:

.NET Core Info

.NET Core version used for development:
.NET Core version installed in the environment where application runs:
Output of dotnet --info:
Contents of project.json/project.csproj:

Thanks, Ashish

Hello @ashishdhingra Error occurs when we add HostedService to our web application. Before all works correct and now log in ClowdWatch exists. Expected Behavior Do not create aws-logger-errors.txt. All logs should be in CloudWatch. Current Behavior Lots of of log generate aws-logger-errors.txt. Content of file you can see above. Possible Solution same issue in stackoverflow Environment AWSSDK.Core version used: 3.5.1.40 Service assembly and version used: AWSSDK.CloudWatch v 3.5.0.40 Operating System and version: Amazon Linux 4.14.138-114.102.amzn2.x86_64 x86_6 Visual Studio version: Microsoft Visual Studio Enterprise 2019 Version 16.5.4 Targeted .NET platform: .NET Core Info : 3.1.404 .NET Core version used for development: 3.1.404 .NET Core version installed in the environment where application runs: 3.1.404 Output of dotnet --info: .NET Core SDK (reflecting any global.json): Version: 3.1.404 Commit: 470f6754b3

Runtime Environment: OS Name: amzn OS Version: 2 OS Platform: Linux RID: linux-x64 Base Path: /usr/share/dotnet/sdk/3.1.404/

Host (useful for support): Version: 3.1.10 Commit: 1721e39439

.NET Core SDKs installed: 3.1.404 [/usr/share/dotnet/sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs: https://aka.ms/dotnet-download

Contents of project.json/project.csproj:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <DocumentationFile>MobileAppAPI.NetCore.API.xml</DocumentationFile>
    <OutputPath></OutputPath>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="IdentityModel" Version="4.1.1" />
    <PackageReference Include="IdentityServer4" Version="3.1.0" />
    <PackageReference Include="IdentityServer4.AccessTokenValidation" Version="3.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.1" />
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.1.3" />
    <PackageReference Include="Quartz" Version="3.0.7" />
    <PackageReference Include="Serilog.AspNetCore" Version="3.2.0" />
    <PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="5.0.0-rc4" />
    <PackageReference Include="Swashbuckle.AspNetCore.Filters" Version="5.0.0-rc8" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\MobileAppAPI.NetCore.Common\MobileAppAPI.NetCore.Common.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Config\MobileAppAPI.NetCore.Config.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Data\MobileAppAPI.NetCore.Data.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Kpi\MobileAppAPI.NetCore.Kpi.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Security\MobileAppAPI.NetCore.Security.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Swagger.UI\MobileAppAPI.NetCore.Swagger.UI.csproj" />
    <ProjectReference Include="..\MobileAppAPI.NetCore.Swagger\MobileAppAPI.NetCore.Swagger.csproj" />
  </ItemGroup>

  <ItemGroup>
    <None Update="MobileAppAPI.NetCore.API.xml">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

  <ProjectExtensions><VisualStudio><UserProperties appsettings_1json__JsonSchema="" /></VisualStudio></ProjectExtensions>
</Project>

Hi @ValeryYegorov,

Just to re-confirm,

Are all the logs written to CloudWatch correctly?
Is the issue really about aws-logger-errors.txt being generated which is growing in size?

If the issue is really about the generation of aws-logger-errors.txt file, then you could try disabling it by setting AWSLoggerConfig.LibraryLogFileName to null.

Thanks, Ashish

Hello Ashish,

CloudWatch does have all the logs, Main issue is aws-logger-errors.txt file is generated; and is growing (as you mentioned).

Would like to know the root cause of it being generated, to ensure we are not losing any REQUIRED logging details.

And for valid exceptions, need to handle them properly, instead of simply disabling it.

Thanks in advance & Regards, Dev

Hi @debabrata789,

The LibraryLogFileName (default value is aws-logger-errors.txt) is the name of the file into which errors from the AWS.Logger.Core library will be written into. It could be due to several reasons (as per my analysis of the code):

The AWS Logger in-memory buffer has reached maximum capacity
Background Task is busy, and not yet claimed the previous release (Maybe busy with credentials)
Some CloudWatch service error while flushing the logs during process shutdown.
Connection issues with the service.

For the exception in your case, refer the Stackoverflow question https://stackoverflow.com/questions/34525955/aws-cloudwatch-invalidsequencetokenexception. There appears to be issue with connecting to CloudWatch service in your case and hence the exception is written to aws-logger-errors.txt.

Hope this helps.

Thanks, Ashish

This issue has not received a response in a week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

aws / aws-logging-dotnet