search

NuGet / NuGetGallery

NuGet Gallery is a package repository that powers https://www.nuget.org. Use this repo for reporting NuGet.org issues.
https://www.nuget.org/
Apache License 2.0
1.55k stars 644 forks source link

[Azure Search] db2azs throws out of memory exception #7231

Closed loic-sharma closed 5 years ago

loic-sharma commented 5 years ago 
2019-06-03 17:00:50 [Information] Fetching packages with package registration key >= 104120 and <= 105220 (~9957 packages).
2019-06-03 17:00:56 [Information] Pushing batch of 1000 to index "search-033".
2019-06-03 17:00:56 [Information] Pushing batch of 1000 to index "search-033".
2019-06-03 17:01:26 [Error] An exception was thrown while processing package ID "Wyam.Docs.Samson".
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Microsoft.Azure.Search.Models.IndexAction.Upload[T](T document)
   at NuGet.Services.AzureSearch.Db2AzureSearch.PackageEntityIndexActionBuilder.GetHijackIndexAction(String packageId, Package package, HijackDocumentChanges changes) in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\PackageEntityIndexActionBuilder.cs:line 170
   at NuGet.Services.AzureSearch.Db2AzureSearch.PackageEntityIndexActionBuilder.<>c__DisplayClass4_0.<AddNewPackageRegistration>b__2(KeyValuePair`2 p) in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\PackageEntityIndexActionBuilder.cs:line 59
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at NuGet.Services.AzureSearch.Db2AzureSearch.PackageEntityIndexActionBuilder.AddNewPackageRegistration(NewPackageRegistration packageRegistration) in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\PackageEntityIndexActionBuilder.cs:line 57
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ConsumeWorkAsync>d__17.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 201
2019-06-03 17:01:26 [Error] An exception was thrown while processing package ID "Fancy.ResourceLinker".
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Linq.Set`1.Resize()
   at System.Linq.Set`1.Find(TElement value, Boolean add)
   at System.Linq.Enumerable.<ExceptIterator>d__73`1.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__14.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 177
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__13.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 103
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushFullBatchesAsync>d__11.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 92
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ConsumeWorkAsync>d__17.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 208
2019-06-03 17:01:26 [Error] An exception was thrown while processing package ID "Vertechcon.DynamicLibraries.Compilable".
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Linq.Set`1.Resize()
   at System.Linq.Set`1.Find(TElement value, Boolean add)
   at System.Linq.Enumerable.<ExceptIterator>d__73`1.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__14.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 171
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__13.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 103
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushFullBatchesAsync>d__11.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 92
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ConsumeWorkAsync>d__17.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 208
2019-06-03 17:01:26 [Error] An exception was thrown while processing package ID "NeinLinq.Queryable".
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Linq.Set`1.Resize()
   at System.Linq.Set`1.Find(TElement value, Boolean add)
   at System.Linq.Enumerable.<ExceptIterator>d__73`1.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__14.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 171
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushBatchesAsync>d__13.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 103
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.BatchPusher.<PushFullBatchesAsync>d__11.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\BatchPusher.cs:line 92
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ConsumeWorkAsync>d__17.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 208
2019-06-03 17:01:26 [Error] "Job Failed to Run": "System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at lambda_method(Closure , Shaper )
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper.HandleEntityAppendOnly[TEntity](Func`2 constructEntityDelegate, EntityKey entityKey, EntitySet entitySet)
   at lambda_method(Closure , Shaper )
   at System.Data.Entity.Core.Common.Internal.Materialization.Coordinator`1.ReadNextElement(Shaper shaper)
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.<MoveNextAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Data.Entity.Infrastructure.IDbAsyncEnumerableExtensions.<ForEachAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.NewPackageRegistrationProducer.<GetPackagesAsync>d__6.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\NewPackageRegistrationProducer.cs:line 123
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.NewPackageRegistrationProducer.<ProduceWorkAsync>d__4.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\NewPackageRegistrationProducer.cs:line 55
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ProduceWorkAsync>d__16.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<PushAllPackageRegistrationsAsync>d__14.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 148
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ExecuteAsync>d__12.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 98
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Services.AzureSearch.Db2AzureSearch.Db2AzureSearchCommand.<ExecuteAsync>d__11.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Services.AzureSearch\Db2AzureSearch\Db2AzureSearchCommand.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at NuGet.Jobs.Job.<Run>d__1.MoveNext() in E:\NuGet.Services.Metadata\src\NuGet.Jobs.Db2AzureSearch\Job.cs:line 23
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Jobs.JobRunner.<JobLoop>d__11.MoveNext()"
loic-sharma commented 5 years ago

False alarm, this was my fault b/c of changes I did for testing. I disabled the BatchPusher's pushing of hijack actions, which caused it to store all hijack actions in memory.