search

chocolatey / choco

Chocolatey - the package manager for Windows
https://chocolatey.org
Other
10.33k stars 903 forks source link

choco outdated.outdated ignore-pinned uses correct enhanced exit codes Pester tests fails in some Test Kitchen scenarios #2801

Closed corbob closed 2 years ago

corbob commented 2 years ago

What You Are Seeing?

When Team City kicks off a Test Kitchen run for a freshly completed Chocolatey component version, and while the Test Kitchen is running a new version of the component is published to internal feeds, this particular test reports a failure:

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26

What is Expected?

This test should not fail, because in this case the package is expected to be outdated.

How Did You Get This To Happen? (Steps to Reproduce)

In this specific instance:

  1. Chocolatey Agent PR was merged to develop branch
  2. Team City kicks off the chained Test Kitchen build
  3. Another Chocolatey Agent PR was merged to develop branch while the Test Kitchen build is running
  4. Test Kitchen fails this specific test
  5. Team City kicks off a chained Test Kitchen build for step 3's version
  6. Test Kitchen succeeds this test (assuming no further merges to develop

System Details

Test Kitchen related, occurs on both 2016 and 2019 kitchens

Output Log

Output from the Team City email:

 Failed tests summary: 2 (2 new)
(new) Pester - <chocolatey-service-tests-windows-2016>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)         details »
(new) Pester - <chocolatey-service-tests-windows-2019>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)         details »

Stacktraces:
(new) Pester - <chocolatey-service-tests-windows-2016>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26

(new) Pester - <chocolatey-service-tests-windows-2019>: choco outdated.outdated ignore-pinned uses correct enhanced exit codes.Exits with correct exit code (0)

Expected 0, because Chocolatey v1.2.0-alpha-20220813-1f34adf2 Business
Outdated Packages
 Output is package name | current version | available version | pinned?
chocolatey-agent|1.1.0-alpha-20220818-67|1.1.0-alpha-20220818-68|false
Chocolatey has determined 1 package(s) are outdated., but got 2.
at $Output.ExitCode | Should -Be $ExitCode -Because $Output.String, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26
at <ScriptBlock>, C:\Users\Administrator\AppData\Local\Temp\verifier\suites\chocolatey-service-tests\chocolatey-licensed-tests\chocolatey-tests\choco-outdated.Tests.ps1:26

Proposed solution: Maintain a list of internal Chocolatey components that are pinned during the setup of these tests. As they sometimes encounter the described scenario, they should rightly be pinned for this particular test.

AdmiringWorm commented 2 years ago

:tada: This issue has been resolved in version 1.2.0 :tada:

The release is available on:

Your GitReleaseManager bot :package::rocket: