SoS Health Check for VMware Cloud Foundation instance - OPERATION_IN_PROGRESS

[antonystefanov]

Code of Conduct

• [X] I have read and agree to the project's Code of Conduct.
• [X] Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
• [X] Do not leave "+1" or other comments that do not add relevant information or questions.
• [X] If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

Generate VCFHealth report:

$sddcManagerFqdn="sfo-vcf01.sfo.rainpole.io"
$sddcManagerUser="administrator@vsphere.local"
$sddcManagerPass="VMw@re1!"
$reportPath="/home/admin/HRMReports"
$sddcManagerLocalPass="VMw@re1!VMw@re1!"
$sddcManagerLocalUser="vcf"

Invoke-VcfHealthReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -sddcManagerLocalUser $sddcManagerLocalUser -sddcManagerLocalPass $sddcManagerLocalPass -allDomains

Fails with error:

Script File:       /home/admin/.local/share/powershell/Modules/PowerVCF/2.4.1.1000/PowerVCF.psm1 Line: 4090
Relevant Command:  $response = Invoke-RestMethod -Method POST -Uri $uri -Headers $headers -ContentType 'application/json' -Body $jsonBody
Target Uri:         https://sfo-vcf01.sfo.rainpole.io/v1/system/health-summary
Exception Message: Response status code does not indicate success: 409 (CONFLICT).
Error Message:     {"errorCode": "OPERATION_IN_PROGRESS", "message": "Operation is in progress for Id 778450b2-b0ff-4467-a7fc-c2b0c5cad261. Wait for the operation to complete."}

 [07-11-2024_10:40:17]   Error at Script Line 1072
 [07-11-2024_10:40:17]   Relevant Command: $response = Get-VCFHealthSummaryTask -id $requestID
 [07-11-2024_10:40:17]   Error Message: Cannot validate argument on parameter 'id'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
 [07-11-2024_10:40:17] INFO Generating the Service Health Report using the SoS output for VMware Cloud Foundation instance ([sfo-vcf01.sfo.rainpole.io](http://sfo-vcf01.sfo.rainpole.io/)).
 [07-11-2024_10:40:17]   Error at Script Line 314
 [07-11-2024_10:40:17]   Relevant Command: $serviceHtml = Invoke-Expression "Publish-ServiceHealth -json $jsonFilePath -html $($failureOnlySwitch)"; $reportData += $serviceHtml
 [07-11-2024_10:40:17]   Error Message: Missing an argument for parameter 'json'. Specify a parameter of type 'System.String' and try again. 

Use Case(s)

Step 1: Run the report generation Invoke-VcfHealthReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -sddcManagerLocalUser $sddcManagerLocalUser -sddcManagerLocalPass $sddcManagerLocalPass -allDomains

Step 2: Then interrupt with CTRL+C

Step 3: Then run again: Invoke-VcfHealthReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -sddcManagerLocalUser $sddcManagerLocalUser -sddcManagerLocalPass $sddcManagerLocalPass -allDomains

Fails with error that SOS is in progress OPERATION_IN_PROGRESS

Step 4: Wait 10+ mins and retry the report generation - completes with success

Potential Configuration

VCF 5.1.1 (GA)

PS /home/admin> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.18
PSEdition                      Core
GitCommitId                    7.2.18
OS                             Linux 5.10.219-1.ph4 #1-photon SMP Wed Jun 26 09:49:07 UTC 2024
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /home/admin> get-module | ft name, version

Name                             Version
----                             -------
Microsoft.PowerShell.Management  7.0.0.0
Microsoft.PowerShell.Utility     7.0.0.0
PackageManagement                1.4.7
PowerShellGet                    2.2.5
PowerValidatedSolutions          2.10.2.1001
PowerVCF                         2.4.1.1000
PSReadLine                       2.1.0
VMware.CloudFoundation.Reporting 2.6.2.1001
VMware.VimAutomation.Common      13.2.0.22643733
VMware.VimAutomation.Core        13.2.0.22643732
VMware.VimAutomation.Sdk         13.2.0.22643730
VMware.vSphere.SsoAdmin          1.3.9

$sddcManagerFqdn="sfo-vcf01.sfo.rainpole.io"
$sddcManagerUser="administrator@vsphere.local"
$sddcManagerPass="VMw@re1!"
$reportPath="/home/admin/HRMReports"
$sddcManagerLocalPass="VMw@re1!VMw@re1!"
$sddcManagerLocalUser="vcf"

Invoke-VcfHealthReport -sddcManagerFqdn $sddcManagerFqdn -sddcManagerUser $sddcManagerUser -sddcManagerPass $sddcManagerPass -reportPath $reportPath -sddcManagerLocalUser $sddcManagerLocalUser -sddcManagerLocalPass $sddcManagerLocalPass -allDomains

References

No response

[tenthirtyam]

Yes, this would happen if cancelled. One option is to lock the option to cancel in the console (like Platform Tools) and another to error handle a little more gracefully.

[tenthirtyam]

@bhumitra - would you mind taking a look at this one and suggest an approach?

[bhumitra]

I looked at this one, if we do a ctrl+c, it cancels the script, but doesnt cancel the operation on SDDC manager.

Enhancements to the PS module -

1. Prompt user when Ctrl+c is pressed and gracefully kill the script by terminating the corresponding operation on SDDC Manager.
2. Do an API call to Check if an existing operation for SOS is running on SDDC manager. IF it is running, provide appropriate response to user - e.g. "An instance of SOS health check is already running on SDDC manager. Please wait until it is completed" If no operation is running, launch the health check. I will work on this enhancement next week.

[github-actions[bot]]

'Marking this issue as stale due to inactivity. This helps us focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!'

[tenthirtyam]

Reassigning to @garlicNova.