search

dataplat / dbatools

🚀 SQL Server automation and instance migrations have never been safer, faster or freer
https://dbatools.io
MIT License
2.39k stars 787 forks source link

Update tests to be able to run them outside of AppVeyor (part 2) #9413

Open andreasjordan opened 6 days ago

andreasjordan commented 6 days ago

Followup to #9406

I want to run all the tests on a local lab using "real world" sql server instances.

These are some fixes to get all -DbaDb tests to run on my local lab against three local instanes without any failures.

andreasjordan commented 5 days ago

For Invoke-DbaWhoisActive.Tests.ps1: Can we add \CommunitySoftware\sp_whoisactive-12.00.zip to the appveyorlabrepo? @potatoqualitee Do you want to add the file from https://github.com/amachanic/sp_whoisactive/archive/refs/tags/v12.00.zip or should I open a pull request to the repo?

andreasjordan commented 5 days ago

I also added some changes to commands to fix some issues to be able to run the tests.

About the change to Connect-DbaInstance: Maybe I have deleted this line some time in the past, I just can not remember. I can move this change to a separate PR for better documentation.

niphlod commented 4 days ago

I don't think adding whoisactive is a problem. I should have enough perms to do it, wait a bit

niphlod commented 4 days ago

done @andreasjordan , check if everything matches

andreasjordan commented 4 days ago

In my local lab, all 699 tests complete in around 90 minutes.

These currently have failing tests, will work on them in the next week:

Add-DbaAgDatabase.Tests.ps1 Add-DbaAgListener.Tests.ps1 Add-DbaAgReplica.Tests.ps1 Copy-DbaAgentJob.Tests.ps1 Copy-DbaLinkedServer.Tests.ps1 Copy-DbaResourceGovernor.Tests.ps1 Export-DbaDacPackage.Tests.ps1 Get-DbaAgDatabase.Tests.ps1 Get-DbaAgListener.Tests.ps1 Get-DbaAgReplica.Tests.ps1 Get-DbaAvailabilityGroup.Tests.ps1 Get-DbaClientProtocol.Tests.ps1 Get-DbaDump.Tests.ps1 Get-DbaLatchStatistic.Tests.ps1 Invoke-DbaBalanceDataFiles.Tests.ps1 Invoke-DbaWhoisActive.Tests.ps1 New-DbaAvailabilityGroup.Tests.ps1 New-DbaLogin.Tests.ps1 New-DbaSsisCatalog.Tests.ps1 Read-DbaAuditFile.Tests.ps1 Read-DbaXEFile.Tests.ps1 Remove-DbaAgDatabase.Tests.ps1 Remove-DbaAgListener.Tests.ps1 Remove-DbaAvailabilityGroup.Tests.ps1 Remove-DbaDatabaseSafely.Tests.ps1 Restore-DbaDatabase.Tests.ps1 Resume-DbaAgDbDataMovement.Tests.ps1 Set-DbaAgReplica.Tests.ps1 Set-DbaAvailabilityGroup.Tests.ps1 Set-DbaErrorLogConfig.Tests.ps1 Set-DbaTcpPort.Tests.ps1 Set-DbaTempDbConfig.Tests.ps1 Start-DbaDbEncryption.Tests.ps1 Suspend-DbaAgDbDataMovement.Tests.ps1 Sync-DbaLoginPermission.Tests.ps1 Test-DbaBackupEncrypted.Tests.ps1 Test-DbaDeprecatedFeature.Tests.ps1 Watch-DbaXESession.Tests.ps1

Update: Updated the list, because some tests passed on the second execution. Update: Updated the list, because some tests passed after the first instance was installed as an express edition.

andreasjordan commented 3 days ago

Test for Invoke-DbaWhoIsActive is failing on instance2:

Describe : Invoke-DbaWhoisActive Integration Tests
Context  : Should have SPWhoisActive installed correctly
Name     : It Should be installed to master
Result   : Failed
Message  : Expected $null or empty, but got @([07:57:48][Install-DbaWhoIsActive] Failed to install stored procedure. | Log entry string is too long. A string written to the event log cannot exceed 32766 characters.).

Let's see if it works on instance3...

andreasjordan commented 3 days ago

@niphlod - sorry, I need to learn a little more about the tests. I changed Invoke-DbaWhoIsActive tests to use instance3 instead of instannce2 to see if it fails on this instance as well. What do I need to changes also to more the test in the other test group?

niphlod commented 3 days ago

hi @andreasjordan , the "logic" that assigns tests to a specific job within the matrix of concurrent jobs in appveyor (we name it as "scenario") is within appveyor.common.ps1.

Trying to sum up all the things involved....

pester.groups.ps1 is the place where "ties are created" between jobs on appveyor and scenarios, based on:

So, as soon as a test has just "instance3" in it, without "instance1", it'll go (or, at least, that's the idea) automatically into the 2016_2017 scenario. Scenarios (run tests with a set of preconfigured services active, see https://github.com/dataplat/dbatools/blob/development/appveyor.yml#L35 and following) are further split on "parts" in order to run tests concurrently. Each scenario gets set up the same way via "init scripts", such as \tests\appveyor.SQL2017.ps1. Each test either lands either on "part1" or "part2" automatically (if parts are present), and they're balanced with a simple count, i.e. the logic tries to keep the number of test files assigned to a Job (which is the "result" of the matrix) even, but still considers an absolute "bound" by scenario to not cross (tests that are assigned to 2016_2017 can't cross the "scenario" bound, just the "part" one).