search

microsoft / contributor-community-experiments

Tracking experiments and sharing best practices that we learn to build strong communities in our GitHub repos
MIT License
5 stars 2 forks source link

Protect tooling experience from regression #5

Open danmoseley opened 3 years ago

danmoseley commented 3 years ago

Hypothesis

Repos have CI to keep their build and tests working, but do not have an automated way to protect the tooling experience, and this allows regressions that significantly impair productivity.

Experiment

Success criteria

HongGit commented 2 years ago

@captainsafia mentioned that ASp.NET Core runs validation on contributions to the repo with VS. It's not automated., the it does catch issues if there are any.

@RussKie @ViktorHofer could you please share what actions are taking regarding this one?

RussKie commented 2 years ago

I'm not quite sure what "tooling experience" means in this context.

Since dotnet/winforms is a UI tech that requires a designer and is tightly coupled with Windows, all out development is generally done in VS (this is why I've been annoying various folks about improving devex for dotnet/* repo in VS). We have several test harness apps in our solution that we run (F5) and use for debugging/validation. This is manual and helps catching regressions (usually from upstream dependencies, such as arcade or runtime). As long as a contribution is sensible and aligns with our overarching roadmap, builds pass and all tests are green - we merge it. We rely on our wonderful CTI test team (and to a degree on our consumers) to report any regressions (be that to Windows Forms runtime, designer, .NET runtime or VS).

danmoseley commented 2 years ago

The idea behind this one is that (at least in my repo) we carefully validate all PR's to ensure they don't break the command line build or the tests of our product. We have next to zero validation that they do not break the tooling of developers working in the repo. For example, PR's have broken any of -- loading in Visual Studio, building, debugging, running tests, etc. We discover this problem later and then have to fix it, but this causes developer pain and particularly so for the community who cannot as easily ask for help or are not as familiar with our infrastructure. What would be ideal is if we can do some basic validation of these scenarios during PR validation. Perhaps something as simple as devenv.com /build would protect solution load and build at least, as a start.

More generally, it would be good to know how other repos protect their own contributor tooling scenarios and what are best practices.