Open torronen opened 3 years ago
All of the failed systems were connected to a specific wifi network, while the ones were not. So, possibly a network issues. If so, could the Internet connectivity requirement be enabled by opt-in, or at least ignore failed connections.
Issue seems re-appears even in offline mode. It would seem to appear randomly - the longer training runs, the higher the chance of getting stopped by JSON-RPC. This is the highest priority issue of the ones our team has logged. Any advice on how to debug the issue or find further information on it? The problem is encountered on new PC's with fresh installs + all updates, most with wifi or offline. The older workstations and Windows Servers with good connectivity did not see this error.
Hi @torronen
Could we have the servicehub log on your machine? It's located in %temp%servicehub folder and is of great help in detecting what's going wrong. According to what you describe of 407 FastTreeOva
, I'm suspecting an out-of-memory error caused by large model + large dataset.
If the log doesn't indicate anything helpful, we might ask you to set ServiceHubTraceLevel
to All
and run again through following steps. But for now let's take a look at existing log first.
ServiceHubTraceLevel
to All
in environmentYes, give me up to a few days to get it.
It could be a RAM issue. The datasets are 1.5 - 2.5 Gb, the computers with the regular issues have 16 gb RAM. There may have been a report from a computer with 80 Gb RAM, too, but I need to confirm it. I believe we may also have had 1 or 2 Visual Studio crash during training ("disappear" with no errors). On other hand, I think some of the computers with 16 Gb have been running fine so far past 5 600 tested models on a 1.8gb dataset. Anyway, you will know more as you receive the log files.
Meanwhile, you can also collect the error information from event viewer, simply by selecting "Windows Logs" => "Application" and search "ServiceHub.Host.CLR.x64.exe"
Actually, my home laptop had just got same error message even if the dataset is different. servicehub.zip It is from today, so you can probably ignore most of the files which have not been updated today
16 Gb RAM, VS 16.10.2, will update now to 16.10.3 and set ServiceHubTraceLevel to ALL and restart training Last iteration: 68 LightGbmMulti 2.3 Gb dataset size, with 2484 columns
Windows Log does not have ServiceHub.Host.CLR.x64.exe but has APPCRASH for Microsoft.ServiceHub.Controller.exe
Here is export of Microsoft.ServiceHub.Controller.exe application log events some-application-logs.zip
I see there are large number of crashes for NahimicSvc32.exe at the same time which is probably the sound card driver for Lenovo. It could be related to the issue since Lenovo Legion laptops and desktops are encountering the issue regularily, but Macbook, older Lenovo laptop or Dell servers did not encounter the problem. It could be a symptom of running out of RAM, but anyway I'll try to disable Nahimic service as well for next run,.
Windows error reporting log files for ServiceHub Controller. windows-appcrash-log-files.zip
Here is from another laptop, about 2Gb dataset with 16 gb RAM. Log level was still standard, nahmic was already disabled, servicehub files were deleted earlier and it seems this error did not write anything there, visual studio was still 16.10.2.
In this case, Windows Application logs showed error for ServiceHub.Host.CLR.x64.exe which is exported to the logs.zip file. Error seems to be running out of virtual memory as you suspected. ServiceHub.Host.CLR.x64.exe is consuming 52,033,503,232 bytes (48.5G), devenv 478,851,072 bytes (~470M).
It seems even multiple laptops are identical, the error occurs irregularily. This one went to 1731. Error probably happened on model 1732?
This leads to questions:
I wonder if I could still recover Model.zip from some temp folder in these cases?
On one of the computers running for a longer time, I see fluctuations of 100 - 2000 page faults/second from ServiceHub.Host.CLR.x64, while RAM consumption fluctuates between 17%-49%, averaging maybe bit above 20%.
Hi @torronen Thanks for the update, ~50GB memory usage sounds to be the main reason why servicehub host crash and JSON-RPC connection lost. I'll try to reproduce it on my machine to confirm it though.
I wonder if I could still recover Model.zip from some temp folder in these cases?
In some way, yes. The model's name is simply the iteration name + .zip, so you can just go to that folder, find out the model you want based on iteration name, and copy and paste it to target folder, if it's not covered by other experiments.
How to estimate the need for RAM or virtual memory?
It's hard to estimate. Most ML.Net trainers are streamed so they shouldn't use a lot of memory during training. I'll reach out to mlnet team to find out how non-streamed trainers works.
Can AutoML skip the models requiring too much memory, or can it at least ignore errors?
It's a bit tricky to do that because the error is caused by servicehub host crash, so modelbuilder doesn't know the memory outage until the crash happens. I'll reach out to servicehub team to see if there's anything we can do here.
If errors can not be ignored, could it somehow keep the last best model? Currently, it seems all progress is lost.
We are working on "continue training", which allows users to pause/continue train at anytime. It should alleviate the progress-loss issue in some way. But it's tricky to keep the last best model for now, because everything from training process lost when the training service crash.
How to estimate the need for RAM or virtual memory?
IDK, I need to reach out to servicehub team to find out the answer.
@torronen another question, do you remember the last trainer when crash happens? Is that all lgbm?
And can you also pack model builder log and upload here? You can find it in %temp%\MLVSTools\logs. It contains every bits of logs during training process, including the content in output window
Thanks for the prompt replies.
yes, I think they are all lgbm, on what I could find. Here is debug files from today, but unfortunately this did not have ServiceHubTraceLevel set to All so the servicehub log folder is probably useless. It is in the zip file anyway.
I believe you mean this file: 46bddea9-59a9-4ae0-9276-cc58947ac50c.txt
Hi @torronen
Since the OOM error possibly only happen on lgbm, we can provide a specifc build that allows you to disable some trainers through environment variables. In the meantime, can you help us verify if the problem goes after lgbm trainer is disabled?
Hi @torronen
The build from the following feed will allow you to disable trainers through the environment. You can enable or disable trainers through env: AutoMLServiceMultiClassifiers
, env: AutoMLServiceBinaryClassifiers
and env: AutoMLServiceRegressors
. The default value for those three variables are all identical: LGBM;RF;FASTTREE;LBFGS;SDCA
. To disable either trainer, you just need to remove it's name from the value string. Here're a few examples
feed: https://privategallery.blob.core.windows.net/gallery/refs/heads/u/xiaoyun/addFFToTrainer/atom.xml
disable LGBM for Multiclass classifiers
env: AutoMLServiceMultiClassifiers
= RF;FASTTREE;LBFGS;SDCA
only enable fast tree for binary classifiers
env: AutoMLServiceBinaryClassifiers
= FASTTREE
enable all trainers for regressors
simply not set the env: AutoMLServiceRegressors
, or env: AutoMLServiceRegressors
= LGBM;RF;FASTTREE;LBFGS;SDCA
You will also need to restart the VS each time you edit environment variables.
thanks, I will try it. I am currently trying if setting a fixed paging file to 368 GB would fix it. So far, potentially promising.
Meanwhile, I might have a new issue of computers restarting / crashing (next time I come I will see they are restarted and waiting for login). Are there additional log files I should look at to see if Visual Studio caused the restart? Windows Event Log does not reveal much to me, although I see there is some COM component crashing. I would like to verify it is about VS or training before opening a new issue.
I did not yet get to run with LGBM disabled, but I had another case where FastForestOva failed. This time ServiceHub.Host.CLR.x64.exe consumes 58.6 Gb. I will try next with lgbm and fastforestova disabled.
ML log file: a5d1f7f7-cdef-43db-907c-3e4306516f24.txt
Windows event log items: case10-event-logs.zip
So, potentially the best approach is to have at least 32x dataset size of RAM available (or possibly, paging file of same size), or maybe it is exponential? BTW, most my datasets are above the 1gb limit (1.3 - 2.5gb) officially supported by model builder according to Github front page. On other hand, I am testing 258 GB dataset on a server with 1TB ram, and passed 191 models tested so far.
This is an interesting case. It is from Windows server w/ EPYC, 164GB RAM and 368 GB page file. Dataset size 2.3Gb. Ended in "JSON-RPC error" due to OOM. Windows log says Servicehub consumed 512 GB at time of resource exhaustion. It did not yet have the debug version of the plugin.
FastTreeOva, last completed model 743. I believe (but can not confirm) same dataset with same settings has progressed much further on a laptop with 16gb. Should I try the same with the AutoML API and skip Model builder to confirm it is not a model builder issue? The much bigger datasets probably have not gone this far as they run slower.
ML log 0059986b-e2b3-4679-9eb8-f5039c2a51ce.txt
Windows log case11-windows-log.zip
Total available memory is about the same as what Servicehub consumed until error
If the memory required is exponential for a given dataset size then it may not be possible to run past 1 gb, but if not maybe we should run a RAM consumption monitor to see if it keeps increasing due to memory leak or something unnecessary (such as earlier models) are kept unnecessarily in RAM?
Temporarily, I think it would be adequate if the best model would be stored in some easy-to-find and copy-to-solution place. Potentially .consumption and .training files could also be generated already every time there is a new best model? Even if OOM happens, the progress would not be lost / could be recovered fairly easily.
Potentially .consumption and .training files could also be generated already every time there is a new best model?
The code is generated after training is success so no it won't be updated everytime a new best model found
Meanwhile, I also find something interesting from lgbm repo, which datasets is ~1gb and it uses up to 150GB of memory during training.
So another potential solution might be just to increase the virtual memory considering that you have ~500GB left on your drive. as a work-around.
Yes, I will do that, make the page file to max size for now. That is adequate solution. However, I am concerned if might keep creating bigger and bigger models until time runs out or all space is consumed. Do you think think it is possible it would keep creating bigger and bigger models?
If bigger model results to better result it will. The main parameters automl service tuning for lgbm is num_leaves, num_iterations and max_bin, where num_leaves and max_bin have a direct infect over the memory usage during training.
Since the servicehub process gets crashed because of using up virtual memory, other than increasing virtual memory or ignore trainers that use too much memory, there's little thing model builder can do here.
In the future, having trials running in a separate process (or several separate processes) other than servicehub should be the right way to go, which makes it possible for model builder to monitor and handle memory exceptions ( instead of being killed by system together with training process).
I get this immediately on adding SQL Server and choosing a table.
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.DataProviders.FileDataProvider.<GetCsvDataProgramAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<SetDatabaseTableAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.TextDataControl.<>c__DisplayClass6_0.<<TableSelectionChanged>b__0>d.MoveNext()
This is the log when trying to a CSV file:
2022-03-21 21:55:01.0631 DEBUG The JSON-RPC connection with the remote party was lost before the request could complete.
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<GetCsvDataProgramAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<SetFilePathAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.DataTextViewModel.<SetFilePathAsync>d__98.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.TextDataControl.<<BrowseFileButton_Click>b__5_0>d.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2022-03-21 21:55:42.1201 DEBUG The JSON-RPC connection with the remote party was lost before the request could complete.
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<GetCsvDataProgramAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<SetFilePathAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.DataTextViewModel.<SetFilePathAsync>d__98.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.TextDataControl.<<BrowseFileButton_Click>b__5_0>d.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
Hi @sjh37
The log itself just says servicehub service was terminated but it doesn't tell why... Could you present us with the schema of the database table you try to add
also @beccamc for notification
@LittleLittleCloud @beccamc I want to use the [Data] Table. I have included both tables for completeness as there is a FK between the two:
CREATE TABLE [dbo].[Data]
(
[Symbol] [varchar] (9) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
[Date] [date] NOT NULL,
[Open] [decimal] (19, 5) NOT NULL,
[HIGH] [decimal] (19, 5) NOT NULL,
[LOW] [decimal] (19, 5) NOT NULL,
[Close] [decimal] (19, 5) NOT NULL,
[Volume] [bigint] NOT NULL,
[NextClose] [decimal] (19, 5) NULL
)
CREATE TABLE [dbo].[DataStatus]
(
[Symbol] [varchar] (9) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
[ProcessingFlag] [int] NOT NULL,
[Info] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
)
GO
ALTER TABLE [dbo].[Data] ADD CONSTRAINT [PK_Data] PRIMARY KEY CLUSTERED ([Symbol], [Date]) ON [PRIMARY]
ALTER TABLE [dbo].[DataStatus] ADD CONSTRAINT [PK_DataStatus] PRIMARY KEY CLUSTERED ([Symbol]) ON [PRIMARY]
GO
ALTER TABLE [dbo].[Data] ADD CONSTRAINT [FK_Data_DataStatus] FOREIGN KEY ([Symbol]) REFERENCES [dbo].[DataStatus] ([Symbol]) ON DELETE CASCADE
GO
However, I get the same error with a CSV file too.
@sjh37 Hmmm if csv file is failing too,, it's probably not the error in sql data source.
If possible could you share with us a copy of failing csv so we can reproduce the error, regards
Meanwhile, it would be helpful if we can have your VS version/model builder version and CPU version as well.
@LittleLittleCloud @beccamc I'm on GMT time, sorry for the delay.
Microsoft Visual Studio Community 2022
Version 17.1.1
VisualStudio.17.Release/17.1.1+32228.430
Microsoft .NET Framework
Version 4.8.04084
Installed Version: Community
.NET Core Debugging with WSL 1.0
.NET Core Debugging with WSL
ADL Tools Service Provider 1.0
This package contains services used by Data Lake tools
ASA Service Provider 1.0
ASP.NET and Web Tools 2019 17.1.358.51495
ASP.NET and Web Tools 2019
ASP.NET Web Frameworks and Tools 2019 17.1.358.51495
For additional information, visit https://www.asp.net/
Azure App Service Tools v3.0.0 17.1.358.51495
Azure App Service Tools v3.0.0
Azure Data Lake Tools for Visual Studio 2.6.5000.0
Microsoft Azure Data Lake Tools for Visual Studio
Azure Functions and Web Jobs Tools 17.1.358.51495
Azure Functions and Web Jobs Tools
Azure Stream Analytics Tools for Visual Studio 2.6.5000.0
Microsoft Azure Stream Analytics Tools for Visual Studio
C# Tools 4.1.0-5.22109.6+0c82c4114a4e4b8b723b915eee3b13261db6717f
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Code alignment 15.1.118.0
Code alignment helps you present your code beautifully, enhancing clarity and readability.
Common Azure Tools 1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
Fabric.DiagnosticEvents 1.0
Fabric Diagnostic Events
File Icons 2.7
Adds icons for files that are not recognized by Solution Explorer
GhostDoc 22.1.22055.0
Generate XML Comments from your code, maintain clean and up-to-date documentation, produce help documentation in multiple formats, use intelligent source code Spell Checker in Visual Studio.
Markdown Editor v2 2.0.86
A full featured Markdown editor with live preview and syntax highlighting. Supports GitHub flavored Markdown.
Microsoft Azure Hive Query Language Service 2.6.5000.0
Language service for Hive query
Microsoft Azure Service Fabric Tools for Visual Studio 17.0
Microsoft Azure Service Fabric Tools for Visual Studio
Microsoft Azure Stream Analytics Language Service 2.6.5000.0
Language service for Azure Stream Analytics
Microsoft Azure Tools for Visual Studio 2.9
Support for Azure Cloud Services projects
Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines
Microsoft Library Manager 2.1.161+abc97ecc7d.RR
Install client-side libraries easily to any web project
Microsoft MI-Based Debugger 1.0
Provides support for connecting Visual Studio to MI compatible debuggers
Microsoft Visual Studio Tools for Containers 1.2
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.
Node.js Tools 1.5.40105.1 Commit Hash:1822ee94b29c6cf748a19825f14cc26d30b0b871
Adds support for developing and debugging Node.js apps in Visual Studio
NuGet Package Manager 6.1.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/
Razor (ASP.NET Core) 17.0.0.2206201+62a2c1d6162f828801565a7ca26d9d48b810a05b
Provides languages services for ASP.NET Core Razor.
SQL Server Data Tools 17.0.62201.12090
Microsoft SQL Server Data Tools
Syntax Visualizer 1.0
An extension for visualizing Roslyn SyntaxTrees.
ToolWindowHostedEditor 1.0
Hosting json editor into a tool window
TypeScript Tools 17.0.1229.2001
TypeScript Tools for Microsoft Visual Studio
Visual Basic Tools 4.1.0-5.22109.6+0c82c4114a4e4b8b723b915eee3b13261db6717f
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Visual F# Tools 17.1.0-beta.21610.4+07b5673e4f2fa7630e78abe37f16b372353a7242
Microsoft Visual F# Tools
Visual Studio Code Debug Adapter Host Package 1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio
Visual Studio Container Tools Extensions 1.0
View, manage, and diagnose containers within Visual Studio.
Visual Studio Extension for SpecFlow 1.0
Visual Studio extension for working with SpecFlow projects and Gherkin feature files.
Visual Studio IntelliCode 2.2
AI-assisted development for Visual Studio.
Visual Studio Tools for Containers 1.0
Visual Studio Tools for Containers
https://user-images.githubusercontent.com/4190514/159676860-58e604d5-f9b4-4d98-9607-b69aae8ebe66.mp4
Hi @sjh37 Thanks for the reply,, could you also share with us servicehub log as well? You can find it under
%temp%servicehub\logs folder and there should be a log file starts or contains with MicrosoftMLModelBuilderAutoML
.
@LittleLittleCloud servicehub_logs.zip
I deleted the folder, started up VS and tried to open a CSV file this time, same error I get when trying to read a SQL table.
This looks so wield, the servicehub log also doesn't contain any useful information other than saying that it's been terminated...
Is that possible for you to provide us with a minimal reproduce dataset if you can get the same error while load a csv file? Regards!
I think I will have to resign myself to uninstalling and re-installing visual studio. However, VS works perfectly fine in every other way.
I don't think VS is the one that blamed,,, Did you try other scenarios successfully or having the same error
Everything that requires a CSV crashes, including Image classification
even though there was no CSV file involved with that.
2022-03-25 13:43:48.9739 INFO Set log file path to C:\Users\simon\AppData\Local\Temp\MLVSTools\logs\MLModel1-FKC3NX.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2022-03-25 13:44:29.4360 DEBUG The JSON-RPC connection with the remote party was lost before the request could complete.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<<StartTrainingAsync>b__97_0>d.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
Hmmm that's super wield... Looks like everything that in servicehub is not working
Can you provide us your OS information, your CPU && if you're running in a VM or not?
I also disabled Jetbrains Resharper and that didn't make a difference, still the same JSON-RPC exception. What remote party is it trying to connect to?
I am not using a VM, but may have to try this option if I can't get it working.
OS Name Microsoft Windows 10 Home
Version 10.0.19044 Build 19044
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name DESKTOP-CP7C32T
System Manufacturer Dell Inc.
System Model XPS 13 9305
System Type x64-based PC
System SKU 0A5C
Processor 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz, 2803 Mhz, 4 Core(s), 8 Logical Processor(s)
BIOS Version/Date Dell Inc. 1.4.1, 23/02/22
SMBIOS Version 3.2
Embedded Controller Version 255.255
BIOS Mode UEFI
BaseBoard Manufacturer Dell Inc.
BaseBoard Product 0PPYW4
BaseBoard Version A02
Platform Role Mobile
Secure Boot State On
PCR7 Configuration Elevation Required to View
Windows Directory C:\Windows
System Directory C:\Windows\system32
Boot Device \Device\HarddiskVolume1
Locale United States
Hardware Abstraction Layer Version = "10.0.19041.1566"
User Name DESKTOP-CP7C32T\simon
Time Zone GMT Standard Time
Installed Physical Memory (RAM) 16.0 GB
Total Physical Memory 15.7 GB
Available Physical Memory 8.28 GB
Total Virtual Memory 31.7 GB
Available Virtual Memory 21.6 GB
Page File Space 16.0 GB
Page File C:\pagefile.sys
Kernel DMA Protection On
Virtualization-based security Running
Virtualization-based security Required Security Properties
Virtualization-based security Available Security Properties Base Virtualization Support, Secure Boot, DMA Protection, UEFI Code Readonly, SMM Security Mitigations 1.0, Mode Based Execution Control, APIC Virtualization
Virtualization-based security Services Configured
Virtualization-based security Services Running
Device Encryption Support Elevation Required to View
A hypervisor has been detected. Features required for Hyper-V will not be displayed.
@sjh37 Chipping in before LittleLittleCloud gets back: I also got confused about this at first, but it seems Servicehub executable and Visual Studio communicate through JSON-RPC. It is not a remote party. The actual training happens in Servicehub process. If it crashes, then VS just says it no longer can connect to this process that is suppose to do the work. In my case, I saw some errors in Windows Event log. If you did not yet have a look at it yet, maybe Windows Event Viewer could give LittleLittleCloud some more info, if we are lucky.
In my case, I ran out of RAM and only fix was to increase size of paging file. Because it ran out of RAM I also had some other device driver crashing in the application log at same time which also confused me a bit at first. You probably should have enough RAM to run at least 100-200 iterations even on a big dataset.
@LittleLittleCloud There were a few errors logged in Event viewer (Thanks @torronen ). I cleared all the event viewer logs with powershell:
wevtutil el | Foreach-Object {wevtutil cl "$_"}
Started VS, and hit the JSON-RPC error. Nothing logged in Event Viewer.
There are slight differences in the error between CSV and SQL:
2022-03-26 22:41:39.3034 DEBUG The JSON-RPC connection with the remote party was lost before the request could complete.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<GetCsvDataProgramAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<SetFilePathAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.DataTextViewModel.<SetFilePathAsync>d__98.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.TextDataControl.<<BrowseFileButton_Click>b__5_0>d.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2022-03-26 22:39:00.5724 DEBUG The JSON-RPC connection with the remote party was lost before the request could complete.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__154.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__143`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.DataProviders.FileDataProvider.<GetCsvDataProgramAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.Actions.DataTextActions.<SetDatabaseTableAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.ToolWindows.TextDataControl.<>c__DisplayClass6_0.<<TableSelectionChanged>b__0>d.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
@LittleLittleCloud Is the issue Simon is experiencing the same root cause?
Since the servicehub process gets crashed because of using up virtual memory, other than increasing virtual memory or ignore trainers that use too much memory, there's little thing model builder can do here.
In the future, having trials running in a separate process (or several separate processes) other than servicehub should be the right way to go, which makes it possible for model builder to monitor and handle memory exceptions ( instead of being killed by system together with training process).
@LittleLittleCloud with the FLAML implementation of AutoML is this still an issue? @torronen @sjh37 would a code sample using the AutoML API help with some of these issues particularly being able to choose which algorithms you use during training?
@luisquintanilla for me, I actually only encountered this in ModelBuilder, but never in Microsoft.ML. Now if Microsoft.ML.AutoML if comparative to ModelBuilder then it is a good alternative. Particularily being able to set the search space to limit consumption of RAM. For sjh37 it was probably not about RAM. If nothing else, I suppose debugging in AutoML API is probably easier than trying to find why servicehub terminates.
@luisquintanilla That would be handy thanks as I would get to use the resources on my laptop instead of having to do this in a VM in Azure.
For clarity, I get the RPC error after browsing for and selecting a CSV file.
I used visual studio installer and removed ML.Net, then re-added it. I then updated the to the latest "Model Builder 2022" VSIX update and got the following error: Install Error : System.IO.IOException: Cannot create a stable subkey under a volatile parent key.
Log:
27/07/22 18:35:48 - Microsoft VSIX Installer
27/07/22 18:35:48 - -------------------------------------------
27/07/22 18:35:48 - vsixinstaller.exe version:
27/07/22 18:35:48 - 17.3.22-preview1
27/07/22 18:35:48 - -------------------------------------------
27/07/22 18:35:48 - Command line parameters:
27/07/22 18:35:48 - C:\Program Files (x86)\Microsoft Visual Studio\Installer\resources\app\ServiceHub\Services\Microsoft.VisualStudio.Setup.Service\VSIXInstaller.exe,C:\Users\simon\Downloads\Microsoft.ML.ModelBuilder.Gallery.vsix,/callingprocessid:43348,/installas:3624
27/07/22 18:35:48 - -------------------------------------------
27/07/22 18:35:48 - Microsoft VSIX Installer
27/07/22 18:35:48 - -------------------------------------------
27/07/22 18:35:50 - Skipping product Microsoft.VisualStudio.Product.BuildTools (853721ee) since it does not support extensions
27/07/22 18:35:50 - Initializing Install...
27/07/22 18:35:50 - Extension Details...
27/07/22 18:35:50 - Identifier : A274A0E2-E41E-40CC-A694-8D5D14BBD2D0
27/07/22 18:35:50 - Name : ML.NET Model Builder 2022
27/07/22 18:35:50 - Author : Microsoft
27/07/22 18:35:50 - Version : 16.13.9.2231505
27/07/22 18:35:50 - Description : Simple UI tool to build custom machine learning models for VS2022.
27/07/22 18:35:50 - Locale : en-US
27/07/22 18:35:50 - MoreInfoURL :
27/07/22 18:35:50 - InstalledByMSI : False
27/07/22 18:35:50 - SupportedFrameworkVersionRange : [4.5,)
27/07/22 18:35:50 -
27/07/22 18:35:55 - SignatureState : ValidSignature
27/07/22 18:35:55 - SignedBy : Microsoft Corporation
27/07/22 18:35:55 - Certificate Info :
27/07/22 18:35:55 - -------------------------------------------------------
27/07/22 18:35:55 - [Subject] : CN=Microsoft Corporation, OU=OPC, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:35:55 - [Issuer] : CN=Microsoft Code Signing PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:35:55 - [Serial Number] : 3300000463CA7009C09247FA42000000000463
27/07/22 18:35:55 - [Not Before] : 02/09/21 19:26:31
27/07/22 18:35:55 - [Not After] : 01/09/22 19:26:31
27/07/22 18:35:55 - [Thumbprint] : 909438EC2298CCB370EEAF0453E440A8DCBBC166
27/07/22 18:35:55 -
27/07/22 18:35:55 - Supported Products :
27/07/22 18:35:55 - Microsoft.VisualStudio.Enterprise
27/07/22 18:35:55 - Version : [17.0,18.0)
27/07/22 18:35:55 - ProductArchitecture : amd64
27/07/22 18:35:55 - Microsoft.VisualStudio.Community
27/07/22 18:35:55 - Version : [17.0,18.0)
27/07/22 18:35:55 - ProductArchitecture : amd64
27/07/22 18:35:55 - Microsoft.VisualStudio.Professional
27/07/22 18:35:55 - Version : [17.0,18.0)
27/07/22 18:35:55 - ProductArchitecture : amd64
27/07/22 18:35:55 -
27/07/22 18:35:55 - References :
27/07/22 18:35:55 - Prerequisites :
27/07/22 18:35:55 - -------------------------------------------------------
27/07/22 18:35:55 - Identifier : Microsoft.VisualStudio.Component.CoreEditor
27/07/22 18:35:55 - Name : Visual Studio core editor
27/07/22 18:35:55 - Version : [15.0,18.0)
27/07/22 18:35:55 -
27/07/22 18:35:55 - Signature Details...
27/07/22 18:35:55 - Extension is signed with a valid signature.
27/07/22 18:35:55 -
27/07/22 18:35:55 - Searching for applicable products...
27/07/22 18:35:55 - Found installed product - Global Location
27/07/22 18:35:55 - Found installed product - ssms
27/07/22 18:35:55 - Found installed product - Visual Studio Community 2022
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES (X86)\COMMON FILES\MICROSOFT\EXTENSIONMANAGER\EXTENSIONS\Microsoft\Windows Kits\10\Desktop SDK\extension.vsixmanifest'
27/07/22 18:35:55 - Capping recursion at 5 directories in each extension root due to 'default limit'.
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES (X86)\COMMON FILES\MICROSOFT\EXTENSIONMANAGER\EXTENSIONS\Microsoft\Windows Kits\10\SDK ARM Additions\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES (X86)\COMMON FILES\MICROSOFT\EXTENSIONMANAGER\EXTENSIONS\Microsoft\Windows Kits\10\Windows Store SDK\extension.vsixmanifest'
27/07/22 18:35:55 - Capping recursion at 5 directories in each extension root due to 'default limit'.
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\3zjnnywu.3rt\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\bi5ls5x0.25q\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\k2pp2znv.uij\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\lyhlbyow.b4u\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\rk1rj2ui.j3r\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\wk0erjns.1s1\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\wq225irz.35g\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\USERS\SIMON\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_9F77A6E2\EXTENSIONS\JetBrains\ReSharperPlatformVs17\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\000kohot.cep\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\03tq5mb0.215\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\0dbdb4jy.y5x\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\13vhzb1d.4zs\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\2q1sha0q.nl5\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\33q0glsd.tbk\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\41wifc2k.ljd\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\bjxb2cxx.wsc\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\bt0deslk.gnj\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\bwajolhn.m4w\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\d1o33dlc.si1\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\euqh1jxr.c3l\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\iskvundv.vyp\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\knxfxkmi.2n1\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\lbmy3jvf.cn5\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\m5qks4h3.e3x\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\mjvtqrqo.rby\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\mmsgsmy1.zxm\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\mytijjck.pys\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\qhszei32.d01\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\ramq4ebc.klq\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\sezohq0p.jjr\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\TestPlatform\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\ucwexzag.qhe\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\uxhxo3uw.gzp\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\VSSDK\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\w3rqlc2e.akg\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\wbhxevnc.cl3\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\wzj2cagj.xfh\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\xspucgwc.alf\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\ycuwub5v.igl\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\yizlges3.pa4\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\zc4mqk3p.o2c\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\zp1vf2sw.lt1\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\ClickOnce\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\ConnectedServices\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\DebugAdapterHost\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\DebuggerServices\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\IntelliCode\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\IntelliCode.CSharp\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\IntelliCode.FindReplaceByExample\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\LiveShare\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\LiveShare.VSCore\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\ManagedProjectSystem\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Microsoft Office 365 Connected Service\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Portable Library Tools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Publish\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\TemplateEngine\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\WCF Connected Service\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Live Preview\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools Shared\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools Diagnostics\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\ADL Tools\2.6.5000.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\AppInsights\Contracts\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\AppInsights\Core\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\CodeSense\Framework\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Modeling\17.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\NodeJsTools\NodeJsTools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\NodeJsTools\TestAdapter\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\SQLDB\SSDTUpdate\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\SQLDB\Vsix\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\SQLEditor\Vsix\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Unit Test Explorer\TestWIExtension\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Visual Studio MPF\10.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Visual Studio MPF\11.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Visual Studio MPF\12.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Visual Studio MPF\14.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web\Mvc\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web\Mvc4\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools\Azure\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools\Library Manager\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools\Common\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools\Deployment Project\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools\RemoteDebugger\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Windows Azure Tools\Resource Manager\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web\Mvc\Scaffolding\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools\Azure\Package\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\EXTENSIONS\Microsoft\Web Tools\Azure\WebJobs\extension.vsixmanifest'
27/07/22 18:35:55 - Capping recursion at 5 directories in each extension root due to 'default limit'.
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\DataDesign\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Asal\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\CallHierarchy\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ClientDiagnostics\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\CodeAnalysis\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\CodingConventions\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Commands\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\CrossRepositorySearch\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\DesignTools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Editor\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ErrorList\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\FSharp\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\HotReload\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\InteractiveWindow\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\LanguageServer\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ModelBuilder\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\NavigateTo\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\NuGet\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\NuGetRecommender\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\OpenFolder\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Performance Tools\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ProductUpdate\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Project\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ProjectServices\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ProjectSystemQuery\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\RazorLanguageServices\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ReferenceManager\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Search\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Terminal\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\TestWindow\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\TextMate\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\TSServer\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\TypeScript\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Windows.Forms\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\XamlDiagnostics\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\XamlLanguageService\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\CPUSampling\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Database\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Debugger\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\DiagnosticsHub\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\DotNetAsync\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\DotNetCounters\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\DotNetObjectAlloc\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\EventsViewer\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\FileIO\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Instrumentation\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\ParallelDebugger\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\PerfMarkers\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Shell\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\TreeNavigation\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Architecture Tools\GraphProviderPackage\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ClientDiagnostics\AppResponsiveness\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Editorconfig\Wizard\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\ExtensionManager\ServiceModule\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\JSDiagnostics\Debugger\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Language\GenerateType\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\TeamFoundation\Team Explorer\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\VBCSharp\ExpressionEvaluators\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\VBCSharp\LanguageServices\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\VBCSharp\SourceBasedTestDiscovery\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\VC\Debugger\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\VisualStudio\Editors\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Instrumentation\IceCap\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\TreeNavigation\GraphProvider\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Platform\Visual Studio MPF\17.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Architecture Tools\Providers\ManagedProvider\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Architecture Tools\Providers\Solution\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Web\Razor\v2.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\Web\Razor\v3.0\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\WebClient\Diagnostics\ChromeDebugAdapter\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\WebClient\Diagnostics\EdgeDebugAdapter\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\WebClient\Diagnostics\Node2DebugAdapter\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\WebClient\Diagnostics\ToolWindows\extension.vsixmanifest'
27/07/22 18:35:55 - Found 'C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\Microsoft\WebClient\Diagnostics\WebDiagnosticsAttach\extension.vsixmanifest'
27/07/22 18:35:56 - The extension will be upgraded from version 16.7.4.2151101.
27/07/22 18:35:57 - BEGIN: Processing extension pack
27/07/22 18:35:57 - END: Processing extension pack
27/07/22 18:35:57 - Initializing Install...
27/07/22 18:35:57 - Extension Details...
27/07/22 18:35:57 - Identifier : A274A0E2-E41E-40CC-A694-8D5D14BBD2D0
27/07/22 18:35:57 - Name : ML.NET Model Builder 2022
27/07/22 18:35:57 - Author : Microsoft
27/07/22 18:35:57 - Version : 16.13.9.2231505
27/07/22 18:35:57 - Description : Simple UI tool to build custom machine learning models for VS2022.
27/07/22 18:35:57 - Locale : en-US
27/07/22 18:35:57 - MoreInfoURL :
27/07/22 18:35:57 - InstalledByMSI : False
27/07/22 18:35:57 - SupportedFrameworkVersionRange : [4.5,)
27/07/22 18:35:57 -
27/07/22 18:36:01 - SignatureState : ValidSignature
27/07/22 18:36:01 - SignedBy : Microsoft Corporation
27/07/22 18:36:01 - Certificate Info :
27/07/22 18:36:01 - -------------------------------------------------------
27/07/22 18:36:01 - [Subject] : CN=Microsoft Corporation, OU=OPC, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:36:01 - [Issuer] : CN=Microsoft Code Signing PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:36:01 - [Serial Number] : 3300000463CA7009C09247FA42000000000463
27/07/22 18:36:01 - [Not Before] : 02/09/21 19:26:31
27/07/22 18:36:01 - [Not After] : 01/09/22 19:26:31
27/07/22 18:36:01 - [Thumbprint] : 909438EC2298CCB370EEAF0453E440A8DCBBC166
27/07/22 18:36:01 -
27/07/22 18:36:01 - Supported Products :
27/07/22 18:36:01 - Microsoft.VisualStudio.Enterprise
27/07/22 18:36:01 - Version : [17.0,18.0)
27/07/22 18:36:01 - ProductArchitecture : amd64
27/07/22 18:36:01 - Microsoft.VisualStudio.Community
27/07/22 18:36:01 - Version : [17.0,18.0)
27/07/22 18:36:01 - ProductArchitecture : amd64
27/07/22 18:36:01 - Microsoft.VisualStudio.Professional
27/07/22 18:36:01 - Version : [17.0,18.0)
27/07/22 18:36:01 - ProductArchitecture : amd64
27/07/22 18:36:01 -
27/07/22 18:36:01 - References :
27/07/22 18:36:01 - Prerequisites :
27/07/22 18:36:01 - -------------------------------------------------------
27/07/22 18:36:01 - Identifier : Microsoft.VisualStudio.Component.CoreEditor
27/07/22 18:36:01 - Name : Visual Studio core editor
27/07/22 18:36:01 - Version : [15.0,18.0)
27/07/22 18:36:01 -
27/07/22 18:36:01 - Signature Details...
27/07/22 18:36:01 - Extension is signed with a valid signature.
27/07/22 18:36:01 -
27/07/22 18:36:01 - Searching for applicable products...
27/07/22 18:36:01 - Found installed product - Global Location
27/07/22 18:36:01 - Found installed product - ssms
27/07/22 18:36:01 - Found installed product - Visual Studio Community 2022
27/07/22 18:36:01 - The extension will be upgraded from version 16.7.4.2151101.
27/07/22 18:36:01 - Initializing Install...
27/07/22 18:36:01 - Extension Details...
27/07/22 18:36:01 - Identifier : A274A0E2-E41E-40CC-A694-8D5D14BBD2D0
27/07/22 18:36:01 - Name : ML.NET Model Builder 2022
27/07/22 18:36:01 - Author : Microsoft
27/07/22 18:36:01 - Version : 16.13.9.2231505
27/07/22 18:36:01 - Description : Simple UI tool to build custom machine learning models for VS2022.
27/07/22 18:36:01 - Locale : en-US
27/07/22 18:36:01 - MoreInfoURL :
27/07/22 18:36:01 - InstalledByMSI : False
27/07/22 18:36:01 - SupportedFrameworkVersionRange : [4.5,)
27/07/22 18:36:01 -
27/07/22 18:36:06 - SignatureState : ValidSignature
27/07/22 18:36:06 - SignedBy : Microsoft Corporation
27/07/22 18:36:06 - Certificate Info :
27/07/22 18:36:06 - -------------------------------------------------------
27/07/22 18:36:06 - [Subject] : CN=Microsoft Corporation, OU=OPC, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:36:06 - [Issuer] : CN=Microsoft Code Signing PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
27/07/22 18:36:06 - [Serial Number] : 3300000463CA7009C09247FA42000000000463
27/07/22 18:36:06 - [Not Before] : 02/09/21 19:26:31
27/07/22 18:36:06 - [Not After] : 01/09/22 19:26:31
27/07/22 18:36:06 - [Thumbprint] : 909438EC2298CCB370EEAF0453E440A8DCBBC166
27/07/22 18:36:06 -
27/07/22 18:36:06 - Supported Products :
27/07/22 18:36:06 - Microsoft.VisualStudio.Enterprise
27/07/22 18:36:06 - Version : [17.0,18.0)
27/07/22 18:36:06 - ProductArchitecture : amd64
27/07/22 18:36:06 - Microsoft.VisualStudio.Community
27/07/22 18:36:06 - Version : [17.0,18.0)
27/07/22 18:36:06 - ProductArchitecture : amd64
27/07/22 18:36:06 - Microsoft.VisualStudio.Professional
27/07/22 18:36:06 - Version : [17.0,18.0)
27/07/22 18:36:06 - ProductArchitecture : amd64
27/07/22 18:36:06 -
27/07/22 18:36:06 - References :
27/07/22 18:36:06 - Prerequisites :
27/07/22 18:36:06 - -------------------------------------------------------
27/07/22 18:36:06 - Identifier : Microsoft.VisualStudio.Component.CoreEditor
27/07/22 18:36:06 - Name : Visual Studio core editor
27/07/22 18:36:06 - Version : [15.0,18.0)
27/07/22 18:36:06 -
27/07/22 18:36:06 - Signature Details...
27/07/22 18:36:06 - Extension is signed with a valid signature.
27/07/22 18:36:06 -
27/07/22 18:36:06 - Searching for applicable products...
27/07/22 18:36:06 - Found installed product - Global Location
27/07/22 18:36:06 - Found installed product - ssms
27/07/22 18:36:06 - Found installed product - Visual Studio Community 2022
27/07/22 18:36:06 - The extension will be upgraded from version 16.7.4.2151101.
27/07/22 18:36:11 - Blocking processes:
27/07/22 18:36:11 - InstallProgressPage ChangeType: 'SingleInstall' operationCount: '2' Text: 'Installing ML.NET Model Builder 2022...'
27/07/22 18:36:11 - The following target products have been selected...
27/07/22 18:36:11 - Visual Studio Community 2022
27/07/22 18:36:11 -
27/07/22 18:36:11 - Beginning to install extension to Visual Studio Community 2022...
27/07/22 18:36:11 - Upgrading 'ML.NET Model Builder', version 16.7.4.2151101 to version 16.13.9.2231505.
27/07/22 18:36:11 - Failed to create scheduled task: UpdateConfiguration_S-1-5-82-271721585-897601226-2024613209-625570482-296978595
27/07/22 18:36:11 - Failed to create the update configuration task for user IIS APPPOOL\.NET v4.5: (9,77):UserId:S-1-5-82-271721585-897601226-2024613209-625570482-296978595
27/07/22 18:36:11 - Failed to create scheduled task: UpdateConfiguration_S-1-5-82-2906086978-416243549-2096465672-1846613839-2027790337
27/07/22 18:36:11 - Failed to create the update configuration task for user IIS APPPOOL\CRMPortal: (9,80):UserId:S-1-5-82-2906086978-416243549-2096465672-1846613839-2027790337
27/07/22 18:36:11 - Failed to create scheduled task: UpdateConfiguration_S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415
27/07/22 18:36:11 - Failed to create the update configuration task for user IIS APPPOOL\DefaultAppPool: (9,79):UserId:S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415
27/07/22 18:36:11 - Failed to create scheduled task: UpdateConfiguration_S-1-5-82-3876422241-1344743610-1729199087-774402673-2621913236
27/07/22 18:36:11 - Failed to create the update configuration task for user IIS APPPOOL\.NET v4.5 Classic: (9,80):UserId:S-1-5-82-3876422241-1344743610-1729199087-774402673-2621913236
27/07/22 18:36:11 - VSIX TaskScheduler: VSIX Update Configuration task has not been successfully registered.
27/07/22 18:36:11 - No mapping between account names and security IDs was done
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.VisualStudio.Debugger.DbgHelp.Win8: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.3]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.VisualStudio.Debugger.Remote.DbgHelp.Win8: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.3]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.VisualStudio.Debugger.Remote.DbgHelp.Win8: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.3]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Windows.UniversalCRT.Msu.8: The current OS Version '10.0.22000.0' is not in the supported version range '[6.2,6.3)'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Windows.UniversalCRT.Msu.81: The current OS Version '10.0.22000.0' is not in the supported version range '[6.3,6.4)'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Net.4.8.FullRedist: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1.1,10.0.17763]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Net.4.8.KB5003306: The current OS Version '10.0.22000.0' is not in the supported version range '[10.0.18362,10.0.18363]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Net.4.8.KB5003304: The current OS Version '10.0.22000.0' is not in the supported version range '[10.0.19041,10.0.19044]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Net.4.8.KB5003793: The current OS Version '10.0.22000.0' is not in the supported version range '[10.0.20279,10.0.20348]'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.DiagnosticsHub.KB2882822.Win7: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.2)'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Windows.UniversalCRT.Msu.7: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.2)'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.VisualStudio.NuGet.PowershellBindingRedirect: The current OS Version '10.0.22000.0' is not in the supported version range '[6.1,6.2)'.
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.Azure.PowerShell: Required conditions are not satisfied: Not Powershell5
27/07/22 18:36:17 - Skipping non-applicable package Microsoft.VisualStudio.OfficeDeveloperTools.WindowsIdentityFoundation.Msu: The current OS Version '10.0.22000.0' is not in the supported version range '(,6.2)'.
27/07/22 18:36:17 - Windows Installer found the package 'Microsoft.NetCore.TargetingPack.5.0.5.0.0,version=17.2.32408.312,chip=x64' superseded but corresponding compatible package information was not found. Attempts to repair will be skipped and that to uninstall may leave the cache behind.
27/07/22 18:36:43 - Install Error : System.IO.IOException: Cannot create a stable subkey under a volatile parent key.
at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str)
at Microsoft.Win32.RegistryKey.CreateSubKeyInternal(String subkey, RegistryKeyPermissionCheck permissionCheck, Object registrySecurityObj, RegistryOptions registryOptions)
at Microsoft.Win32.RegistryKey.CreateSubKey(String subkey, RegistryKeyPermissionCheck permissionCheck)
at VSIXInstaller.WritableSettingsStore.CreateOrOpenWritableCollection(String collectionPath)
at VSIXInstaller.WritableSettingsStore.CreateCollection(String collectionPath)
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.InstalledExtensionStored.SaveHeaderData(IWritableSettingsStore writableSettings) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\InstalledExtensionStored.cs:line 795
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.InstalledExtensionStored.SaveExtensionData(IWritableSettingsStore writableSettings) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\InstalledExtensionStored.cs:line 740
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ExtensionSettingsCache.SaveExtensionToCache(IInstalledExtension extension, CacheMode updateMode) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\ExtensionSettingsCache.cs:line 729
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ExtensionSettingsCache.ProcessAndSaveExtension(IInstalledExtension extension, CacheMode updateMode) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\ExtensionSettingsCache.cs:line 550
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ExtensionSettingsCache.AddOrUpdate(IInstalledExtension extension, CacheMode updateMode) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\ExtensionSettingsCache.cs:line 110
at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.InstalledExtensionSettingsList.Add(IInstalledExtension extension, CacheMode updateMode) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\Settings\InstalledExtensionSettingsList.cs:line 91
at Microsoft.VisualStudio.ExtensionManager.InstallModule.CommitInstalledAndUninstalledExtensions(IEnumerable`1 installedExtensions, IEnumerable`1 uninstalledExtensions, IEnumerable`1 packComponentChanges, Version targetedVSVersion, InstallFlags installFlags) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\EngineModules\InstallModule.cs:line 534
at Microsoft.VisualStudio.ExtensionManager.InstallModule.BeginInstall(IInstallableExtension installableExtension, InstallFlags installFlags, AsyncOperation asyncOp, Version targetedVsVersion) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\EngineModules\InstallModule.cs:line 403
at Microsoft.VisualStudio.ExtensionManager.InstallModule.InstallWorker(IInstallableExtension extension, InstallFlags installFlags, AsyncOperation asyncOp) in D:\a\_work\1\s\src\product\extensionManager\Microsoft.VisualStudio.ExtensionEngine\Impl\EngineModules\InstallModule.cs:line 894
@sjh37 I sent an email to VS team for the installation issue, and I'll put the reply in this thread once I get it.
@sjh37 I reach out to VS team and the installation problem appears to be an old bug in VS2022, and they suggest one of the following fixes that I just quote below
This is an old bug that has since been resolved in 17.3 preview 2. If you have installed a later build, I recommend upgrading to 17.3 prev2 or later, or barring that, delete c:\Program Files (x86)\Microsoft Visual Studio\Installer, and then start any vs install to download a fixed installer.
Could you advise on finding a temporary fix for this problem? Is it related to my Internet connectivity / "phoning home", or a call between processes inside the system?
System Information (please complete the following information):
Describe the bug
To Reproduce Can not be systematically reproduced.
Screenshots If applicable, add screenshots to help explain your problem.![image](https://user-images.githubusercontent.com/26261427/123623820-907e2b80-d816-11eb-9c01-6713b41a0181.png)
Additional context 1 of the computers had last iteration: 407 FastTreeOva ( no note about others ) It seems some of the computers also give a stack overflow notification related to a background task.
Log file can be delivered, but I am concerned it from a different run? Systems have wifi connection to broadband router in case it is a call to the Internet.
End of file snippet: