Question: Shared state file to improve RAR task performance

This is not a performance issue rather it is a question on how to use the MS Build API to solve a performance issue I have.

Issue Description

My build has 7000~ directories in a packages folder. The total time for RAR across the projects in the build is 30 minutes. I currently inject many search paths via SearchPaths into the RAR task for compatibility reasons within my build.

I desire to have a faster build so optimizing RAR felt like a good place to start. A solution I've thought about for a long time is to have a single state file for all of the projects as there is so much duplicated information that each RAR task discovers as it is a bit of a gold fish on a clean CI build.

I noticed in a recent drop of MSBuild that some new RAR properties had shown up with the words "precompute" and "statefile" which sounds very exciting.

A sample was constructed to have a single RAR instance walk the packages directory and hopefully crack open every single DLL in the folder and write that info the cache file which I can then provide to each project.

I've assumed I am able to use AssemblyInformationCacheOutputPath and then feed that back into AssemblyInformationCachePaths.

public class PrecomputeReferenceCache : Task {

        public string[] ReferencePath { get; set; }

        public override bool Execute() {
            System.Diagnostics.Debugger.Launch();

            // 7000 items
            var directories = Directory.GetDirectories(ReferencePath[0], "*", SearchOption.AllDirectories);

            var sw = Stopwatch.StartNew();
            ResolveAssemblyReference rar = new ResolveAssemblyReference() {
                BuildEngine = this.BuildEngine,
                SearchPaths = directories,
                Assemblies = new ITaskItem[] {
                    new TaskItem("WebGrease, Version=1.6.5135.21930, Culture=neutral, PublicKeyToken=31bf3856ad364e35")
                },
                AssemblyInformationCacheOutputPath = @"C:\temp\rar.cache2",
               StateFile = @"C:\temp\rar.cache",
            };

            rar.Execute();
            sw.Stop();
            var elapsed1 = sw.ElapsedMilliseconds;

            sw = Stopwatch.StartNew();
            ResolveAssemblyReference rar2 = new ResolveAssemblyReference()
            {
                BuildEngine = this.BuildEngine,
                SearchPaths = directories,
                Assemblies = new ITaskItem[] {
                    new TaskItem("WebGrease, Version=1.6.5135.21930, Culture=neutral, PublicKeyToken=31bf3856ad364e35")
                },
                AssemblyInformationCachePaths = new ITaskItem[]
                {
                    new TaskItem(@"C:\temp\rar.cache2")
                },
                StateFile = @"C:\temp\rar.cache"

            };
            rar2.Execute();
            sw.Stop();
            var elapsed2 = sw.ElapsedMilliseconds;

            return !Log.HasLoggedErrors;
        }

elapsed1: 4528 elapsed2: 5773

The second RAR instance takes longer than the first instance and still grovels into each folder in SearchPaths.

The output files from the first RAR instance have some size to them

I was hoping that upToDateLocalFileStateCache would contain all of the metadata for each assembly so I can reuse this information for each project and avoid the repeated analysis cost.

What I've learned It appears RAR will only store metadata for a DLL if it is present in the Assemblies list. In my example I'm looking for WebGrease which only appears in a single folder of the 7000.

If I add 7000 unique entries to the Assemblies list will I get the behaviour I desire?

If I cannot get the precomputed state file to work I could look at removing as many SearchPaths as possible to speed up RAR.

Versions & Configurations

Microsoft (R) Build Engine version 16.11.0+0538acc04 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

16.11.0.36601

Regression?

Hi @michael-baker,

Thanks for trying this out! The precomputed cache isn't intended to fill upToDateLocalFileStateCache, as we don't actually have evidence it is up-to-date. In fact, when building your own projects, it may end up invalidating a large fraction of your cache if you create the precomputed cache then modify all your other assemblies and rebuild them.

The precomputed cache should provide an initial value for instanceLocalFileStateCache instead. I noticed from your screenshot that its count was 0, which suggests the cache isn't actually being read in at all. Since rar.cache exists, the most likely explanation is that you had created an empty rar.cache at some point with assembly information for 0 files but correctly formatted—unless this is happening with the second RAR execution, in which case this isn't right. That cache takes precedence over the precomputed cache, so it would completely ignore it when the second RAR iteration came around. As a second (long shot) option, if you happened to use a different version of MSBuild to make the cache then use it, that might have prevented it from being read in properly. We changed the serialization format recently, which would have meant doing work then giving up and defaulting to an empty cache. 35kb also sounds really small; making a cache for a very simple app (what comes out of dotnet new console) is > 1800kb. Does WebGrease depend (directly or independently) on all of your other assemblies? It looks like it took 4.5 seconds to make the cache, whereas to do all the work before took 30 minutes; I expect that you'd have a much larger (and hopefully more useful) cache if you did add all of the assemblies to the list.

When I was testing this while first implementing the feature, I got great numbers like improving RAR execution time by a third, but I've gotten worse results more recently with our new serialization method and format; I'm still looking into that, so I can't 100% promise this cache will be a panacea, just FYI.

Last thought—how many processors do you have? In addition to caching certain information by file, we cache information in processes, and that takes priority, so if most of the assemblies are duplicated between projects, that might mean that using any kind of a file cache means more time reading files and deserializing their contents but no time saved because of the process cache. We also have a longer-term initiative to make a separate RAR-as-a-Service node that would hold all this cached information without having to either deserialize anything more than once or calculate the same assembly information more than once. Even so, initially calculating the paths will still benefit from having fewer search paths, so that isn't a bad direction to go.

Thanks for the help. I'll clarify my goal and approach. I have a full build that takes hours and hours - of which 30 minutes is RAR.

The code is my playground to find a way to reduce cost of RAR by asking it to precompute a humongous state file at the start of the build that contains data for 10,000+ dependencies. I realize that there will be a deserialization cost of such a humongous state file but I'm hoping that the aggressive caching of RAR will mitigate this.

My plan is this

clone repo
get dependencies, packages etc
precompute humongous state file
pass location of humongous state file as global property to each project
build project 1....n

All my my testing is based on the code above - just a a simple MSBuild project that calls a custom C# task with two RAR tasks inside of it. I realize this is a not the intended way that tasks should be composed but I'm trying just experimenting at the moment.

The number of processors doesn't matter as I am controlling the RAR task invocation in my custom task.

The 4.5 seconds is the cost of a single RAR instance to traverse 7000~ directories in the SearchPaths set. If my build contains 10,000+ projects I do not need 10,000 RAR instances all discovering the same information project-by-project.

The state files in in the screenshot are from the playground code.

The first RAR instance creates the two files, which I then just feed into the second instance. I don't do any reflection or anything left field - just using the 'public' API of the RAR task as coded above. Any serialization issues, versioning etc isn't occurring as it all occurs within the same MSBuild process, instance and version.

Only a handful of my actual projects depend on WebGrease. In practice I don't care what inputs I have to feed the precompute task, my goal was to have RAR to traverse all 7000 directories and crack each DLL open it comes across and cache the metadata. It appears RAR seems to ignore caching data for misses. If it goes into a directory that cotnains 50 DLLs and cracks each one but one of them was not WebGrease then it will not populate the cache. At least this is what I've observed by poking around in the debugger.

I'm hoping that the aggressive caching of RAR will mitigate this.

What part of RAR's caching strategy mitigates serialization costs? The caches I'm familiar with are for assembly information, but I think we always read in a state file if it exists for every RAR execution.

All my my testing is based on the code above - just a a simple MSBuild project that calls a custom C# task with two RAR tasks inside of it. I realize this is a not the intended way that tasks should be composed but I'm trying just experimenting at the moment.

I noticed something that confirmed my earlier hunch: the timestamp on rar.cache. It was made earlier, which means it was present both with the first RAR execution in your task and the second. RAR ignores a precomputed cache if there's a not precomputed cache already present. That explains why the precomputed cache had no effect but doesn't explain why it took longer to build the second time than the first. If you forced RAR out of process somehow, maybe it landed on a process with an empty cache the second time? Maybe it was just noise? The presence of the precomputed cache should not have affected the second execution at all, and it should have overwritten rar.cache, so that's two mysteries I might want to dig into.

The number of processors doesn't matter as I am controlling the RAR task invocation in my custom task.

Right; I was thinking for your actual build.

The first RAR instance creates the two files, which I then just feed into the second instance.

RAR should only create one file per execution, and given the timestamp different I noted above, I suspect rar.cache already existed.

It appears RAR seems to ignore caching data for misses. If it goes into a directory that cotnains 50 DLLs and cracks each one but one of them was not WebGrease then it will not populate the cache. At least this is what I've observed by poking around in the debugger.

This is what I would expect. Our caches are basically Dictionary<path, information>, which is great if we know where a file is already. It would be nice if we could have a Dictionary<assemblyName, information including path>, but that would start to have trouble if we ever want to use multiple assemblies. What might be interesting is a fast track for if your search paths (including the order in which you search them) is identical between two times when you're looking for a single assembly. Unless it's right next to referencing projects, you should find the same assembly, so we might be able to speed that up. I'm a little worried we'd break something, but it's certainly an interesting idea.

dotnet / msbuild