search

microsoft / TemplateStudio

Template Studio accelerates the creation of new WinUI 3, WPF, and UWP apps using a wizard-based experience.
Other
2.7k stars 460 forks source link

Core tests are flaky #4421

Open mrlacey opened 2 years ago

mrlacey commented 2 years ago

Describe the bug

Some of the Minimum Core tests do not produce consistent results. It appears to be that the ones that don't always pass are reliant on the file system.

To Reproduce

  1. Run Minimum tests a few times
  2. See errors

Additional context

Follow on from https://github.com/microsoft/TemplateStudio/pull/4417#issuecomment-1083073959

Applies to the following platforms:

This potentially applies to all as the impacted tests (and functionality) are part of the "core" logic

About your setup

X-Ref: https://github.com/microsoft/TemplateStudio/pull/4417#issuecomment-1083073959

mrlacey commented 2 years ago

The flaky tests have been disabled for now so as not to block the release of the initial VS2022 extensions. I'm leaving this open and assigned to me to pick up in the future.

mrlacey commented 2 years ago

Flaky tests are reliant on IO as they interact with the filesystem and files are not always written or unlocked as fast as the tests expect. This seems to be more of an issue with VS2022 & on Win11

mrlacey commented 2 years ago

The cause of the IO-related failures is that there are multiple tests that are creating, copying to, and deleting from directories with the same name. When tests are run individually this isn't a problem. However, when tests are run concurrently, this can be a big issue. Even when run serially, this can still be a problem, as there can be very slight delays in the IO operations returning and changes on disk being completed. This can be enough for artifacts from one test to interfere with the next.

As most of the affected tests are inherited from CoreTemplateStudio, I'm reluctant to do any major rewrites of the tests, however, I think a suitable compromise will be to have each test use different directories (based on the test name). This is a small risk of this causing further complications as it will add to path lengths and so could lead to MAX_PATH issues but they can be addressed separately.