Log events post failed with " Index was outside the bounds of the array"

[ChiefLegend]

I built an image use dockerfile myself from branch "feature-elastic7" several days ago, and run it in Linux-arm64. I can create account and add new project . But there are some error when i try to post events to exceptionless service:

[06:11:20 INF] Processing post: id=a0e7c9ad2a3043a2bc9394f18889f1e0 path=q/442/442f73672b834d82b6c9d968f2d9b429.payload project=5e1e859d340b4200010d2e32 ip=::ffff:10.161.67.47 v=2 agent=exceptionless/4.0.0
[06:11:20 FTL] Error processing action: SaveEventAction Message: Index was outside the bounds of the array.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Microsoft.Extensions.Logging.LogValuesFormatter.GetValue(Object[] values, Int32 index)
   at Microsoft.Extensions.Logging.FormattedLogValues.GetEnumerator()+MoveNext()
   at Serilog.Extensions.Logging.SerilogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
   at Foundatio.Repositories.Elasticsearch.Extensions.LoggerExtensions.LogErrorRequest(ILogger logger, Exception ex, IElasticsearchResponse elasticResponse, String message, Object[] args)
   at Foundatio.Repositories.Elasticsearch.Configuration.Index.CreateIndexAsync(String name, Func`2 descriptor)
   at Foundatio.Repositories.Elasticsearch.Configuration.DailyIndex.EnsureDateIndexAsync(DateTime utcDate)
   at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase`1.IndexDocumentsAsync(IReadOnlyCollection`1 documents, Boolean isCreateOperation, ICommandOptions options)
   at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase`1.AddAsync(IEnumerable`1 documents, ICommandOptions options)
   at Exceptionless.Core.Pipeline.SaveEventAction.ProcessBatchAsync(ICollection`1 contexts)
[06:11:20 ERR] Error processing EventPost a0e7c9ad2a3043a2bc9394f18889f1e0 q/442/442f73672b834d82b6c9d968f2d9b429.payload: Index was outside the bounds of the array.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Microsoft.Extensions.Logging.LogValuesFormatter.GetValue(Object[] values, Int32 index)
   at Microsoft.Extensions.Logging.FormattedLogValues.GetEnumerator()+MoveNext()
   at Serilog.Extensions.Logging.SerilogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
   at Foundatio.Repositories.Elasticsearch.Extensions.LoggerExtensions.LogErrorRequest(ILogger logger, Exception ex, IElasticsearchResponse elasticResponse, String message, Object[] args)
   at Foundatio.Repositories.Elasticsearch.Configuration.Index.CreateIndexAsync(String name, Func`2 descriptor)
   at Foundatio.Repositories.Elasticsearch.Configuration.DailyIndex.EnsureDateIndexAsync(DateTime utcDate)
   at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase`1.IndexDocumentsAsync(IReadOnlyCollection`1 documents, Boolean isCreateOperation, ICommandOptions options)
   at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase`1.AddAsync(IEnumerable`1 documents, ICommandOptions options)
   at Exceptionless.Core.Pipeline.SaveEventAction.ProcessBatchAsync(ICollection`1 contexts)

My elasticsearch version is 7.5.1 and run in the same enviroment. But it doesnot seems have problems. Please let me know which other information you need and how I can obtain it.

[niemyjski]

I did a code review of that code and I don't see anything that would cause this behavior. @ejsmith can you also please take a look.

[ChiefLegend]

I did a code review of that code and I don't see anything that would cause this behavior. @ejsmith can you also please take a look.

I also deploy service on window7 and linux-x86_64 for comparing what different behavior it has. However, it work if exceptionless service run in windows7. And then, to my suprise, service deployed on linux-arm64 work, too!

After repeated tests, I found the error will happen under the following conditions: Empty elasticsearch and deploy exceptionless service on linux.

Here is a log of one of the tests. exceptionless.log

Before 07:46, all new, but post failed (post id 9e5c58c887414a5386be271b25f1ce1f). Then i write a log events to the same ES through another exceptionless service deployed on windows7. It is success. After that, i post log events to the first one again(post id a366de3bc5b94c1a91e82027e136978c). It is success.

[niemyjski]

Thanks for the log, the only thing I can think of is maybe this is a bug in the elastic 5.x library. Is there any chance you could try this with the Exceptionless 6.x docker images running on elastic 7.5.1 to see if you still see this issue on linux. This is the next version of which we currently have in prod and will be going out in the next major release.

[ChiefLegend]

Actually, image used before is built from Exceptionless 6.x ( "feature-elastic7" branch ) . Elastic version was also 7.5.1. The only defference maybe is that i compiled it on windows and wrote dockerfile myself.

[niemyjski]

Is there any chance you can debug this and let us know what the message value and number of arguments is. This should be really easy if you break on this exception type and enable not my code (we have it all setup to pull in dependency source code)

[ChiefLegend]

Sure, but it may be offered in two weeks because we are on Spring Festival holiday.

[ChiefLegend]

Sorry, this is my mistake. I didn't check the two configs between Linux and Windows carefully.

The reason for this problem is that i have not downloaded "mapper-size" plugin for ES. It responded error "root mapping definition has unsupported parameters ..... [_size:{enabled=true}]....." when exceptionless tried to create prod-events-xxxx index. After that, foundatio.repositories.elasticsearch tried to log this error but somehow it failed.

Add "enable-size-plugin=false" into Elasticserach ConnectionString or add mapper-size plugin can avoid this problem.

[niemyjski]

Thanks for the heads up! We are getting close to a release (images are currently rc)