Huge memory usage from reference lookup

[d3nd3]

Hi

https://github.com/oleg-shilo/syntaxer.core/blob/33be172f3dc570f6e61d68b6dcde761ac622fb6c/Intellisense/Autocompleter.cs#L103

In the cs-script-sublime project, when I use : //css_ref UnityEngine.dll

It causes that each time I type UnityEngine. , and the list is populated, 100MB of ram is used. Deleting the . and typing it again, uses another 100MB, so it additively runs out of RAM, before the GC even kicks in. Then it fails with out of memory error.

I don't think this is expected behavior, because I tested a stand-alone program that calls MetadataReference.CreateFromFile and ProjectInfo.Create and the memory usage is only around ~5MB for Creating the metadata an extra time.

Where is the 100MB coming from?

I was using dotnet 7.0 x86, and if I update the cs-script version with latest, it stops working, same with the syntaxer, so I leave them at their defaults.

If the memory usage is normal, I request that it triggers a GC upon re-triggering the . list ? Its currently un-usable in this state for me.

[oleg-shilo]

Hm..., interesting. The only difference between the previous and the latest version of syntaxer is that it is rebuilt for .NET8. There were no code changes at all.

Two questions?

• How can I reproduce it? I don't develop on Unity. How can I prepare a test project/script for your scenario?
• When you detected 100M, did the come just after the refs.AddRange(references.Select(a => MetadataReference.CreateFromFile call?

[d3nd3]

How can I reproduce it? I don't develop on Unity. How can I prepare a test project/script for your scenario?

You'd need UnityEngine.dll which is easy to find inside the Unity Hub directory after installing Unity.\

Its a ~5MB file.

When you detected 100M, did the come just after the refs.AddRange(references.Select(a => MetadataReference.CreateFromFile call?

Yes line 103, specifically was in the stack trace. Each time I lookup the UnityEngine symbols via typing . , there was Resolve() InitWorkSapce() CreateFromFile(), then a Hunk Alloc func, saying out of memory, I think when it reaches like 1400MB of memory, cos of 2GB limit in x86? I am guessing.

I currently switched over to using a LSP solution for the time being.

Shall I confirm that it bugs with 1 .cs file in a directory only?

[oleg-shilo]

Thank you. I will install Unity and try to reproduce it.

Yes please, can you check if it is a matter of the solution of even a single script with no dependency causing the problem? It will help us to understand if I can accurately reproduce it in my environment.

[d3nd3]

Ok say I was able to re-produce it with:

//css_ref "C:\Program Files\Unity\Hub\Editor\2022.3.28f1\Editor\Data\Managed\UnityEngine.dll";

public class Program
{
    UnityEngine
}

the folder contains 1 file , that only. I go to view->console in sublime 4. Btw, is that mb related? That I am using sublime 4 and not 3?. Idk.

Write the dot, and remove the dot about 15-20 times.

send_completion_request
Error: cannot get C# completion from the syntax server
System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb)
   at System.Reflection.Internal.StreamMemoryBlockProvider.ReadMemoryBlockNoLock(Stream stream, Int64 start, Int32 size)
   at System.Reflection.PortableExecutable.PEReader..ctor(Stream peStream, PEStreamOptions options, Int32 size)
   at Microsoft.CodeAnalysis.ModuleMetadata.CreateFromStream(Stream peStream, PEStreamOptions options)
   at RoslynIntellisense.Autocompleter.<>c.<InitWorkspace>b__3_0(String a) in D:\dev\Galos\syntaxer.core\Intellisense\Autocompleter.cs:line 103
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at RoslynIntellisense.Autocompleter.InitWorkspace(AdhocWorkspace workspace, String code, String file, String[] references, IEnumerable`1 includes) in D:\dev\Galos\syntaxer.core\Intellisense\Autocompleter.cs:line 103
   at RoslynIntellisense.Autocompleter.Resolve(String code, Int32 position, String[] references, IEnumerable`1 includes) in D:\dev\Galos\syntaxer.core\Intellisense\Autocompleter.cs:line 854
   at RoslynIntellisense.Autocompleter.GetAutocompletionFor(String rawCode, Int32 position, String[] references, IEnumerable`1 includes, Boolean includDocumentation) in D:\dev\Galos\syntaxer.core\Intellisense\Autocompleter.cs:line 635
   at Syntaxer.SyntaxProvider.GetCompletionRaw(String scriptFile, Int32 caret, Boolean includDocumentation)
   at Syntaxer.SyntaxProvider.GetCompletion(String script, Int32 caret, Boolean includDocumentation) in D:\dev\Galos\syntaxer.core\SyntaxProvider.cs:line 500
   at Syntaxer.SyntaxProvider.ProcessRequest(Args args) in D:\dev\Galos\syntaxer.core\SyntaxProvider.cs:line 117

And cs-script.py in Sublime Packages has been patched to reference C:\Program Files (x86)\dotnet\dotnet.exe instead of system dotnet

[oleg-shilo]

The dot is a trigger. And it does the right job. Pops up the suggestion list when you are about to type a member name. 100M is what concerns me.

I am installing Unity right now.

[oleg-shilo]

Where does this dll come from? I installed Unity tools through VisualStudio Installer and the deployment is different to yours:

Do I need to install something else? My system has no UnityEngine.dll.

[oleg-shilo]

You may try to share UnityEngine.dll here. If it's loadable on its own then I will be able to reproduce the problem. Not to run the script of course.

[d3nd3]

UnityEngine.zip

[oleg-shilo]

Thank you, with the assembly you shared I managed to reproduce the problem. Well, trhe behavier :)

In my test it was 60M jump which corresponded to 50 .NET shared assemblies (~1M per ssembly). In your case the forlder (C:\Program Files\dotnet\shared) with shared assemblies may juts have more assemblies so it jums by 100M.

The behavier itself is not invalid. Since Roslyn requires shared assemblies to be available during the compilation all of them need to be added to the on-fly project as MetadataReference.

However, of course, it's no fun to stress memory that much so I changed the algorithm to cache the shared assemblies in the syntaxer process. And now the consumption is within the expected range: less than 5M (UnityEngine.dll only).

I only need to ensure to the ck the Linux implementation and then can release the fix.

If you do not want to wait, you can use the attached copy. Just ensure that Windows does not lock the assembly when you extract it.

Thank you for your help with reproducing this problem.

syntaxer.v3.1.4.zip

[oleg-shilo]

I just published the release: https://github.com/oleg-shilo/syntaxer.core/releases/tag/v3.1.4.0

But it will take some time before it appears on Chocolatey.