search

microsoft / scalar

Scalar: A set of tools and extensions for Git to allow very large monorepos to run on Git without a virtualization layer
MIT License
1.39k stars 63 forks source link

Enable Trace2 logging of functional tests #482

Closed jeffhostetler closed 3 years ago

jeffhostetler commented 3 years ago

Enable Trace2 logging in Git for the functional test suite. Then zip up the logs and store in as a single artifact.

jeffhostetler commented 3 years ago

This PR causes the CI builds to each temporarily archive about 35Mb (per target) of Trace2 data. I have it set to expire them in 3 days.

Wondering if this should be an optional feature (as in should we have a feature flag defaulting to false at the top and we can add a temporary commit to turn it on when desired) (or if Actions has an official way of doing this).

A quick scan of the Trace2 output shows that the functional tests generates over 200 Git errors that are (I'm assuming) silently being ignored, such as ambiguous argument errors, unknown branch errors, and nonexistent blobs. So we should set aside some time to dig into them at some point and make sure that the C# tests are valid.

derrickstolee commented 3 years ago

This PR causes the CI builds to each temporarily archive about 35Mb (per target) of Trace2 data. I have it set to expire them in 3 days.

Wondering if this should be an optional feature (as in should we have a feature flag defaulting to false at the top and we can add a temporary commit to turn it on when desired) (or if Actions has an official way of doing this).

I thought we had a way of storing the trace2 logs only when the build fails, so we could use the traces to diagnose the problems. (This might only be in the Azure Pipelines builds for VFS for Git.) Would this conditional export be good enough?

A quick scan of the Trace2 output shows that the functional tests generates over 200 Git errors that are (I'm assuming) silently being ignored, such as ambiguous argument errors, unknown branch errors, and nonexistent blobs. So we should set aside some time to dig into them at some point and make sure that the C# tests are valid.

Most of our functional tests compare the behavior in a vanilla Git repo with a Scalar repo. The tests continue to pass if they behave the same. It might be good to do an audit, but who has the time? Something to consider.

jeffhostetler commented 3 years ago

@derrickstolee I didn't see any way to get the trace2 logs here. It may have been on Azure, but I'm not sure any more. Last year I added code to log the C# unit test name and other identifying info as T2 data strings to let us sync up the output; that may be what you're thinking about. This past summer I was working on adding code here to actually capture the logs as artifacts, but got context-switched away before I could finish it.

WRT only logging on failures, I'm interested in both cases. The failures for obvious reasons. And the successes let me look at the timings and the amount and kind of data being returned. That is, I can see if fsmonitor was actually invoked and if it returned non-trivial results to the parent Git command. I could see later adding post processing steps to this script to measure and/or verify some of this, but that's later.