search

pnp / PnP-Tools

Scripts and tools for Office 365 and SharePoint - More for IT Pro's
MIT License
508 stars 394 forks source link

System.OutOfMemoryException error when running the visio scanner tool #261

Open hsugianto opened 3 years ago

hsugianto commented 3 years ago

We are running into a System.OutOfMemoryException error when running the SharePoint Visio Scanner Tool ( https://github.com/pnp/PnP-Tools/tree/master/Solutions/SharePoint.Visio.Scanner). We have attempted to run on the servers where it has larger memory (32gb), running against smaller set of sites, and reducing the threads parameter from the default of 10 to 5 with the same error. Have anyone run into similar behavior? Any pointer that you can provide would be much appreciated.

Full error information:

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.ReadKeyValue() at Microsoft.SharePoint.Client.JsonReader.ReadDictionary() at Microsoft.SharePoint.Client.JsonReader.ReadDictionary[T]()

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Microsoft.SharePoint.Client.JsonReader.Read[T]() at Microsoft.SharePoint.Client.JsonReader.ReadList[T]() at Microsoft.SharePoint.Client.Search.Query.ResultTable.InitOnePropertyFromJson(String peekedName, JsonReader reader) at Microsoft.SharePoint.Client.ClientValueObject.FromJson(JsonReader reader) at Microsoft.SharePoint.Client.JsonReader.ReadJsonObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.Read[T]() at Microsoft.SharePoint.Client.JsonReader.ReadList[T]() at Microsoft.SharePoint.Client.ClientValueObjectCollection1.InitOnePropertyFromJson(String peekedName, JsonReader reader) at Microsoft.SharePoint.Client.Search.Query.ResultTableCollection.InitOnePropertyFromJson(String peekedName, JsonReader reader) at Microsoft.SharePoint.Client.ClientValueObject.FromJson(JsonReader reader) at Microsoft.SharePoint.Client.JsonReader.ReadJsonObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.Read[T]() at Microsoft.SharePoint.Client.ClientResult1.Microsoft.SharePoint.Client.IFromJson.FromJson(JsonReader reader) at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream) at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse() at Microsoft.SharePoint.Client.ClientRequest.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientRequest.d0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientRuntimeContext.d0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContext.d4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContextExtensions.d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount, Int32 delay, String userAgent) at SharePoint.Scanning.Framework.ScanJob.ProcessQuery(Web web, String keywordQueryValue, List1 propertiesToRetrieve, List1 sites, KeywordQuery keywordQuery) in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 381 at SharePoint.Scanning.Framework.ScanJob.Search(Web web, String keywordQueryValue, List1 propertiesToRetrieve, Boolean trimDuplicates) in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 359 at SharePoint.Visio.Scanner.VisioScanJob.ResolveAddedSites(List1 addedSites) at OfficeDevPnP.Core.Framework.TimerJobs.TimerJob.Run() at SharePoint.Scanning.Framework.ScanJob.Execute() in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 118 at SharePoint.Visio.Scanner.VisioScanJob.Execute() at SharePoint.Visio.Scanner.Program.Main(String[] args)

cshin80 commented 3 years ago

Mine ran for around 12 days on a server, then got the memory exception, but without the extensive stack trace, followed by several timout errors before freezing out.

hsugianto commented 3 years ago

Mine ran for around 12 days on a server, then got the memory exception, but without the extensive stack trace, followed by several timout errors before freezing out.

Thank you for the comment and providing your experience running the tool. Our Microsoft DSE had reached out to the author of the visio scanner tool and learned that it's an older open source tool that is no longer maintained with the suggestion to recompile it to 64bit version, limit sites with -r parameter, and possibly exclude the search piece to alleviate the performance overhead. I don't think the tool is meant for scanning a large tenant as I was able to successfully running it in our POC with only about 500 sites without issue.

In the meantime and as the workaround, we use search's graph API to find and search the .vsdw file extension. I hope that helps for your scenario as well.