search

dotnet / fsharp

The F# compiler, F# core library, F# language service, and F# tooling integration for Visual Studio
https://dotnet.microsoft.com/languages/fsharp
MIT License
3.93k stars 788 forks source link

.NET Framework test runs are _super_ slow in CI #17353

Open brianrourkeboll opened 5 months ago

brianrourkeboll commented 5 months ago

(Put this under https://github.com/dotnet/fsharp/labels/Contributor%20Pain, I guess.)

When did the Windows desktop (.NET Framework) build+test runs start taking ~3× as long as the other build+test runs? Was this an expected outcome of some recent change?

The Linux, MacOS, WindowsNoRealsig_testCoreclr, etc., runs take about half an hour now, while the WindowsNoRealsig_testDesktop and WindowsCompressedMetadata desktop_release runs now take more than an hour and a half.

I feel like they didn't take that long even just a couple months ago, either absolutely or relatively. I remember them taking maybe 55 minutes or something (although Azure Pipelines runs older than 30 days have been purged, so I guess I can't prove it).

Screenshot 2024-06-26 153023_highlighted

vzarytovskii commented 5 months ago

There are more of them on desktop. Fsharpqa and Cambridge suites are framework only (not 100% sure about the latter though).

I also won't be surprised if we run VS tests as part of desktop framework, which are very (VERY) slow.

I wanna say it's kinda expected, but nothing we can't fix one way or another

brianrourkeboll commented 5 months ago

There are more of them I guess. Fsharpqa and Cambridge suites are framework only

Yeah, I do remember the desktop runs always taking longer, but I remember them taking ~55 minutes until recently, whereas they now take ~90 minutes.

T-Gro commented 4 months ago

I have looked at this and have already cut it down by ~30 minutes in the past, but over 18 months it grew back without many tests added.

The time comes from the FSharp.Suite tests, which are spinning up a lot of processes and IO operations per each test (creating a project, compiling via fsc.exe, running the built project, sometimes also running peverify.exe on it).

My assumption is that IO got overall slower on the machines due to security features being added, but that is just an assumption.

The resolution is to migrate tests over from this File + Process heavy suite to inmemory tests, ideally just the typecheck function in ComponentTests (i.e. not compile and not verifyIl if not absolutely needed)

vzarytovskii commented 4 months ago

Everything should ideally be moved under component tests (and then we can easily separate them via "--filter" in the CI if needed).