Attempting to increase reliability of some Pester tests

HowardWolosky commented 4 years ago

Description

We're seeing some inconsistent failures in some of the Pester tests.

The hypothesis is that GitHub may need a little bit more time after the creation of objects before performing certain operations on them (like renaming repos), or may need more time after deleting them before it will successfully return a 404 on a successive Get request.

I have added a number of Start-Sleep's throughout the test codebase wherever we've encountered inconsistent failures, and that appears to have resolved the problem. We may need to continue to do more of these if additional ones pop up.

The duration of the sleep itself is controlled by $script:defaultSleepSecondsForReliability which is defined in Tests/Common.ps1.

Long term, I have opened #267 which poses the idea of switching over to mocking out Invoke-WebRequest for the majority of the tests, and instead focus on validating the data that it's sending matches the expected values per the API documentation, and having just a limited number of tests that do actual end-to-end testing.

Issues Fixed

Fixes #264

References

n/a

Checklist

[ ] ~~You actually ran the code that you just wrote, especially if you did just "one last quick change".~~
[ ] ~~Comment-based help added/updated, including examples.~~
[x] Static analysis is reporting back clean.
[x] New/changed code adheres to our coding guidelines.
[ ] ~~Formatters were created for any new types being added.~~
[ ] ~~New/changed code continues to support the pipeline.~~
[ ] ~~Changes to the manifest file follow the manifest guidance.~~
[x] Unit tests were added/updated and are all passing. See testing guidelines. This includes making sure that all pipeline input variations have been covered.
[ ] ~~Relevant usage examples have been added/updated in USAGE.md.~~
[ ] ~~If desired, ensure your name is added to our Contributors list~~

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

HowardWolosky commented 4 years ago

/azp run PowerShellForGitHub-CI

azure-pipelines[bot] commented 4 years ago

Azure Pipelines successfully started running 1 pipeline(s).

microsoft / PowerShellForGitHub