This PR contains a couple of changes to improve the E2E test stability and robustness. The "specs" tests run the E2E tests currently, but later they will be also used for "system tests" (see #62).
There are a couple of improvements included in this PR, but the most important is that it provides a solution for using cached packages. This issue we have discovered with @livioc on #54: If you modify the generation part (e.g. MsTestV2GeneratorProvider) and rerun the related "specs" test, the test will pick up the previously generated nuget package from the package cache and therefore the test will not reflect your changes (you still see the old behavior).
The reason for this is that these E2E test use the generated NuGet packages for generating the sample projects. Between two compilations these packages have the same version number, therefore if a version has been used, nuget automatically caches that in the local nuget cache (C:\Users\<your-user>\.nuget\packages), so the subsequent execution will re-use the cached version. The solution was to manually clean the Reqnroll related packages from the local nuget cache.
There has been tries to avoid this problem: in SpecFlow we tried to generate commit specific nuget versions (this caused other problems and did not solve the changes within commits) and also there was a code that tried to delete the packages from the local nuget cache (most of the cases did not work because of file locking issues).
This PR uses a different approach: it reconfigures nuget for the test projects to use an alternative local nuget cache folder somewhere in $TEMP, so that the execution always starts with an empty nuget cache. (There is a related issue in nuget where this idea was mentioned.) The nuget cache has several megabytes, so instead of making a local nuget cache for every sample project, we introduce a "Test Run" folder, that contains all sample project folders and the local nuget cache. Each time a new test run folder is created in $TEMP, but we have a cleanup mechanism that removes old test run folders.
I have tested that by replaying the changes of #54 and it works fine, all changes of the code are immediately reflected in tests.
Other improvements in this PR:
For those tests that only assert for the total number of tests we also verify that those were passing (in #54 many problems were hidden because of that).
The TestProjectGenerator related projects are now target netstandard2.0 and net6.0 only.
The obsolete -a argument of nuget has been replaced
Improve "specs" to assert for a successful compilation except otherwise stated (in #54 we also had this) and provide better error message
Added some additional helper methods to TestProjectGenerator that will be needed for "system tests"
Get rid of the warnings in the generated Log.cs files
This PR contains a couple of changes to improve the E2E test stability and robustness. The "specs" tests run the E2E tests currently, but later they will be also used for "system tests" (see #62).
There are a couple of improvements included in this PR, but the most important is that it provides a solution for using cached packages. This issue we have discovered with @livioc on #54: If you modify the generation part (e.g.
MsTestV2GeneratorProvider) and rerun the related "specs" test, the test will pick up the previously generated nuget package from the package cache and therefore the test will not reflect your changes (you still see the old behavior).The reason for this is that these E2E test use the generated NuGet packages for generating the sample projects. Between two compilations these packages have the same version number, therefore if a version has been used, nuget automatically caches that in the local nuget cache (
C:\Users\<your-user>\.nuget\packages), so the subsequent execution will re-use the cached version. The solution was to manually clean the Reqnroll related packages from the local nuget cache.There has been tries to avoid this problem: in SpecFlow we tried to generate commit specific nuget versions (this caused other problems and did not solve the changes within commits) and also there was a code that tried to delete the packages from the local nuget cache (most of the cases did not work because of file locking issues).
This PR uses a different approach: it reconfigures nuget for the test projects to use an alternative local nuget cache folder somewhere in $TEMP, so that the execution always starts with an empty nuget cache. (There is a related issue in nuget where this idea was mentioned.) The nuget cache has several megabytes, so instead of making a local nuget cache for every sample project, we introduce a "Test Run" folder, that contains all sample project folders and the local nuget cache. Each time a new test run folder is created in $TEMP, but we have a cleanup mechanism that removes old test run folders.
I have tested that by replaying the changes of #54 and it works fine, all changes of the code are immediately reflected in tests.
Other improvements in this PR:
-aargument of nuget has been replacedTypes of changes