With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code

Steps to reproduce

Convert a proprietary codebase (which I can't share, my apologies).
Wait a couple of hours
In phase 2, simplification, it crashes.

Error message shown

The crash is

https://github.com/dotnet/roslyn threw an exception: System.InvalidOperationException: Exception of type 'System.InvalidOperationException' was thrown.
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.VerifyIntegrity()
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.Add(TKey key, TValue value)
   at Microsoft.CodeAnalysis.GreenNode..ctor(ObjectReader reader)

Full log at https://gist.github.com/jnnnnn/2b7a739170467e125d0a05b0a263774a (with filenames hidden, sorry)

Details

VS extension version 8.5.0.0 (installed 2022-04-14)

Did you see it working in a previous version, which? no
I can follow reasonable requests to debug, but can't share the code it is crashing on

Thanks for the report (and sharing as much info as you could manage)

The roslyn formatter's/simplifier's memory usage scales very badly with file size. So for designer forms which end up as very large files (especially after conversion), assuming your RAM isn't being fully used you might want to try either VS2022, or the command line which are 64-bit and hence can use more RAM.

Someone else had success in a similar situation here: https://github.com/icsharpcode/CodeConverter/issues/605#issuecomment-673959130 You may find enough linked info to workaround it by splitting larger files or by using the command line.

Unfortunately I haven't found time to workaround the issue within the tool yet. The plan is to split files into partial classes and/or format/simplify smaller chunks at a time, and possibly use memoryfailpoint to detect when things are about to go wrong and take evasive action

@GrahamTheCoder, thank you for supporting this tool! I am trying to convert a project with ~2k-3k files. Phase 1 completes smoothly, but in Phase 2 there is one file that refuses to be simplified:

Public Class Common
#Region "Constructor"
    Private Sub New()
    End Sub
#End Region     'Constructor

#Region "Constants"

#If CLR2 Then
    'HKCR path
    private const DataRegistryKey as String = "Infragistics\NetAdvantage\Net\Full\WinForms\CLR2x\Version" & Infragistics.Shared.AssemblyVersion.MajorMinor & "\WinDataDir"
#Else
    'HKCR path
    Private Const DataRegistryKey As String = "Infragistics\NetAdvantage\Net\Full\WinForms\CLR1x\Version" & Infragistics.Shared.AssemblyVersion.MajorMinor & "\WinDataDir"
#End If

#End Region  'Constants

#Region "DataPath"
    ' <summary>
    ' Path to the data installed by the install.
    ' </summary>
    Public Shared ReadOnly Property DataPath() As String
        Get
            Dim dataRegKey As Microsoft.Win32.RegistryKey = Microsoft.Win32.Registry.ClassesRoot.OpenSubKey(DataRegistryKey)
            Dim path As String = Nothing

            If Not dataRegKey Is Nothing Then
                path = CType(dataRegKey.GetValue(Nothing), String)
                dataRegKey.Close()
            End If

            Return path
        End Get
    End Property
#End Region   'DataPath
End Class

There's been no progress for hours.

I am using VS2022 and my machine has 32GB RAM.

Thanks for getting in touch. If it's just that single file that's really interesting! Does the same happen if you right click the file in solution explorer and convert just that one? Note that the process is parallel, so there may be some earlier (much larger) file still being simplified too at the same time.

While it's making no progress, what's the CPU/RAM usage like? Maybe leave task manager open on the performance tab for 30 seconds and send a screenshot of the little graphs down the side just to give a general overview.

If it doesn't repro with just the one class, I don't suppose there's any chance that you're able to share the whole codebase to test with? It'd be great to have a real repro for this.

Note that in the past, the command line (see readme for details) has worked for people in similar situations...though I don't know why, I suspect VS has a special implementation/configuration of some cache or other.

Hi @GrahamTheCoder, sorry for the late response it takes a long time to convert the project that I'm working on.

It looks like it's just that one file that I sent that won't convert. I tried to convert the whole project with both the command line and VS 2022 - the project conversion stalled on just that one file (Common.vb). If I right-click on that file and convert it the same thing happens (stalls at the simplification stage).

Generally, this is what my task manager shows with no apps open besides either VS2022 or the windows command prompt.

Do you have any other recommendations for how to proceed?

Sorry I missed your reply. Thanks for the information. It looks unlikely to be memory related in this case then! The CPU load looks like it is probablby attempting a large single-threaded operation.

I tried creating a project with just the file you posted (and then created one other file defining Infragistics.Shared.AssemblyVersion.MajorMinor to make it compile). When I converted that it went through basically instantly. So it's either system-dependant, or related to the processing of other files too (even the single file conversion uses the context of other files around it)

The next thing I'd suggest is to run from source. At the point when it's been well and truly stuck for a few minutes, if you pause the debugger and grab a screenshot of the parallel stacks window, that should be a big help in identifying the area causing the issue. You may also be able to look in the locals window at which bit of code is being simplified specifically. If you are able to share the whole project with me either publicly or privately I can try to repro/debug this myself (though I understand this may not be possible).

In terms of working around the issue to try to get you some useful output, I've pushed a branch called "skip-simplification". If you check that out and run it. In the instance where it's running, set the option called "Tools->Options->Code Converter->Comment and formatting timeout (minutes)" to something like 1 and it will also apply to the simplification phase (which it doesn't on master). Hopefully that will let it get to the end and write out the files (some of which presumably won't be simplified so will have extra qualifiers / casts / parentheses etc.)

I have confirmed this. When I started the conversion of an over 80000+ lines of code project, it hangs at this file: https://github.com/Aptivi/Kernel-Simulator/blob/4a7c852d7f25886e134605ce3f6462736de08e91/Kernel%20Simulator/Shell/Shells/UESH/UESHShellCommon.vb

All files were converted except this, so I had to use an alternative converter to finish it. Then, I had to manually edit everything so it points to .csproj instead of .vbproj.

During the hang, my processor, Intel Core i7-8700, had 100% usage on one of the 12 cores.

@EoflaOE thanks, that example is likely going to be a massive help! I'll try to repro with it as soon as I can find time and hopefully will finally be able to make progress on this issue 😊

Thanks @EoflaOE this definitely seems to repro the issue even in total isolation. I've created a draft PR with an initial workaround which skips files that take longer than the configurable timeout. Hopefully I'll find a way to improve the performance by calling roslyn in a different way (or possibly making a fix to roslyn)

Here's a snippet of C# which takes more than a minute to simplify (though admittedly contains about 10K characters in a single expression.


using System;using System.Collections;using System.Collections.Generic;using System.Data;using System.Diagnostics;

// Kernel Simulator  Copyright (C) 2018-2022  Aptivi
// 
// This file is part of Kernel Simulator
// 
// Kernel Simulator is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// 
// Kernel Simulator is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
// 
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <https://www.gnu.org/licenses/>.
using global::System.IO;using System.Linq;using System.Net;using System.Xml.Linq;using ColorSeq;using Extensification.DictionaryExts;using Extensification.StringExts;using FluentFTP;using FluentFTP.Helpers;using KS.ConsoleBase;using KS.ConsoleBase.Colors;using KS.ConsoleBase.Inputs;using KS.ConsoleBase.Themes;using KS.ConsoleBase.Themes.Studio;using KS.Files;using KS.Kernel;using KS.Languages;using KS.Login;using KS.Misc.Platform;using KS.Misc.Probers;using KS.Misc.Threading;using KS.Misc.Writers.ConsoleWriters;using KS.Misc.Writers.DebugWriters;using KS.Misc.Writers.FancyWriters;using KS.Shell;using KS.Shell.ShellBase;using KS.Shell.ShellBase.Commands;using KS.Shell.ShellBase.Shells;using global::KS.Shell.Shells.UESH.Commands;using Microsoft.VisualBasic.Constants;using Newtonsoft.Json;using Renci.SshNet;

namespace KS.Shell.Shells.UESH
{
    public static class UESHShellCommon
    {

        internal readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> ModCommands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>();

        /// <summary>
        /// List of commands
        /// </summary>
                                                                                                                                                                                                                                                                public readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> Commands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>() { { "adduser", new global::KS.Shell.ShellBase.Commands.CommandInfo("adduser", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds users", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<userName> [password] [confirm]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.AddUserCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "alias", new global::KS.Shell.ShellBase.Commands.CommandInfo("alias", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds aliases to commands", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { $"<rem/add> <{global::System.String.Join("/", global::System.Enum.GetNames(typeof(global::KS.Shell.ShellBase.Shells.ShellType)))}> <alias> <cmd>" }, true, 3), new global::KS.Shell.Shells.UESH.Commands.AliasCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "arginj", new global::KS.Shell.ShellBase.Commands.CommandInfo("arginj", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Injects arguments to the kernel (reboot required)", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Arguments separated by spaces]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ArgInjCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "beep", new global::KS.Shell.ShellBase.Commands.CommandInfo("beep", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Beep in 'n' Hz and time in 'n' milliseconds", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<37-32767 Hz> <milliseconds>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.BeepCommand()) }, { "blockdbgdev", new global::KS.Shell.ShellBase.Commands.CommandInfo("blockdbgdev", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Block a debug device by IP address", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<ipaddress>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.BlockDbgDevCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "calc", new global::KS.Shell.ShellBase.Commands.CommandInfo("calc", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calculator to calculate expressions.", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<expression>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalcCommand()) }, { "calendar", new global::KS.Shell.ShellBase.Commands.CommandInfo("calendar", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calendar, event, and reminder manager", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<show> [year] [month]", "<event> <add> <date> <title>", "<event> <remove> <eventid>", "<event> <list>", "<event> <saveall>", "<reminder> <add> <dateandtime> <title>", "<reminder> <remove> <reminderid>", "<reminder> <list>", "<reminder> <saveall>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalendarCommand()) }, { "cat", new global::KS.Shell.ShellBase.Commands.CommandInfo("cat", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Prints content of file to console", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-lines|-nolines] <file>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CatCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Wrappable) }, { "cdbglog", new global::KS.Shell.ShellBase.Commands.CommandInfo("cdbglog", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Deletes everything in debug log", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.CdbgLogCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chattr", new global::KS.Shell.ShellBase.Commands.CommandInfo("chattr", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes attribute of a file", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<file> +/-<attributes>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChAttrCommand()) }, { "chdir", new global::KS.Shell.ShellBase.Commands.CommandInfo("chdir", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes directory", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<directory/..>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChDirCommand()) }, { "chhostname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chhostname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes host name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<HostName>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChHostNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chlang", new global::KS.Shell.ShellBase.Commands.CommandInfo("chlang", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes language", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-alwaystransliterated|-alwaystranslated|-force] <language>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChLangCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmal", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmal", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MAL, the MOTD After Login", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMalCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmotd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmotd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MOTD, the Message Of The Day", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMotdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "choice", new global::KS.Shell.ShellBase.Commands.CommandInfo("choice", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Makes user choices", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-o|-t|-m|-a] [-multiple|-single] <$variable> <answers> <input> [answertitle1] [answertitle2] ..." }, true, 3), new global::KS.Shell.Shells.UESH.Commands.ChoiceCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.SettingVariable) }, { "chpwd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chpwd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes password for current user", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<Username> <UserPass> <newPass> <confirm>" }, true, 4), new global::KS.Shell.Shells.UESH.Commands.ChPwdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chusrname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chusrname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes user name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<oldUserName> <newUserName>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChUsrNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "clearfiredevents", new global::KS.Shell.ShellBase.Commands.CommandInfo("clearfiredevents", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears all fired events", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClearFiredEventsCommand()) }, { "cls", new global::KS.Shell.ShellBase.Commands.CommandInfo("cls", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears the screen", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClsCommand()) } };

    }
}

cc @Nortus222 if https://github.com/icsharpcode/CodeConverter/pull/942 lands, then you should talk to @stfedwards about converting our projects from VB to C# since we previously failed due to this bug.

Thanks Alex,

We actually just got the code converted yesterday with a workaround for one file which stopped the conversion. We are currently in the process if review the conversion and fix the couple thousand errors.

I copied Phil who did the actual conversion if you would like to ask him any questions.

Sincerely,

Stephan Edwards, Marketplace Software Office: (949) 288-3956 Cell: 808-306-0437

@.***https://emanageone.com/ Serious Software for Serious Business

From: Stephen (Alex) Wallen @.> Sent: Thursday, August 25, 2022 1:01 PM To: icsharpcode/CodeConverter @.> Cc: Stephan Edwards (e-manage|ONE) @.>; Mention @.> Subject: Re: [icsharpcode/CodeConverter] With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code (Issue #877)

cc @Nortus222https://github.com/Nortus222 if #942https://github.com/icsharpcode/CodeConverter/pull/942 lands, then you should talk to @stfedwardshttps://github.com/stfedwards about converting our projects from VB to C# since we previously failed due to this bug.

— Reply to this email directly, view it on GitHubhttps://github.com/icsharpcode/CodeConverter/issues/877#issuecomment-1227836851, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKJSSMUADXSBULB45LBWT2TV273LLANCNFSM5TMP4Q2A. You are receiving this because you were mentioned.Message ID: @.**@.>>

icsharpcode / CodeConverter