Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.86k forks source link

`Invoke-AzContainerInstanceCommand` doesn't wait for executed command completion in Azure DevOps pipeline Azure PowerShell task #19247

Open andysbolton opened 2 years ago

andysbolton commented 2 years ago

Description

When I run this command in an Azure Devops pipeline with an ubuntu-20.04 image and the Azure Powershell Task, I notice that it it returns immediately and continues running the pipeline script. If I run locally, it blocks for 10 seconds before returning.

Invoke-AzContainerInstanceCommand `
        -ContainerGroupName $Name `
        -ContainerName $Name `
        -ResourceGroupName $ResourceGroup `
        -Command "sleep 10"

Because of this, I'm not able to determine in the pipeline whether the executed command has succeeded or failed.

(FYI I also created an issue in the azure-pipelines-task repo: https://github.com/microsoft/azure-pipelines-tasks/issues/16590)

I'm not sure if this is because the pipeline task is run in NonInteractive mode (although the above command works in NonInteractive mode locally).

If I execute a command that writes to standard out and then put in a Start-Sleep command in the pipeline task, I receive all of the standard out, so it seems the websockets are working correctly, but it is not waiting for the executed command to finish.

Should this command be expected to block in the pipeline or is there some limitation in the pipeline/task or the PowerShell library that is preventing that from happening?

Issue script & Debug output

(Apologies for the garbled output, it's due to the pipeline obfuscating secrets)


$DebugPreference='Continue'
Invoke-AzContainerInstanceCommand `
        -ContainerGroupName $Name `
        -ContainerName $Name `
        -ResourceGroupName $ResourceGroup `
        -Command "sleep 10"

2022-08-16T15:33:18.2963070Z [***;1mDEBUG: 15:***:18 - GetAzureRMContextCommand end processing.[***m
2022-08-16T15:33:18.3262710Z [***;1mDEBUG: [CmdletBeginProcessing]: Starting command[***m
2022-08-16T15:33:18.3314054Z [***;1mDEBUG: CmdletBeginProcessing: [***m
2022-08-16T15:33:18.3316538Z [***;1mDEBUG: CmdletProcessRecordStart: [***m
2022-08-16T15:33:18.3318778Z [***;1mDEBUG: CmdletGetPipeline: [***m
2022-08-16T15:33:18.3321274Z [***;1mDEBUG: CmdletBeforeAPICall: [***m
2022-08-16T15:33:18.3325320Z [***;1mDEBUG: URLCreated: /subscriptions/***/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***/containers/***/exec?api-version=2***21-***9-***1[***m
2022-08-16T15:33:18.3329333Z [***;1mDEBUG: RequestCreated: /subscriptions/***/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***/containers/***/exec?api-version=2***21-***9-***1[***m
2022-08-16T15:33:18.3332084Z [***;1mDEBUG: HeaderParametersAdded: [***m
2022-08-16T15:33:18.3336841Z [***;1mDEBUG: BodyContentSet: [***m
2022-08-16T15:33:18.3370878Z [***;1mDEBUG: ============================ HTTP REQUEST ============================
2022-08-16T15:33:18.3371818Z 
2022-08-16T15:33:18.3373710Z HTTP Method:
2022-08-16T15:33:18.3375329Z POST
2022-08-16T15:33:18.3375826Z 
2022-08-16T15:33:18.3377496Z Absolute Uri:
2022-08-16T15:33:18.3380516Z https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***/containers/***/exec?api-version=2***21-***9-***1
2022-08-16T15:33:18.3381697Z 
2022-08-16T15:33:18.3383283Z Headers:
2022-08-16T15:33:18.3385506Z x-ms-unique-id                : 2
2022-08-16T15:33:18.3387952Z x-ms-client-request-id        : ***d6ea***6f-c812-456e-859c-9***6df11d2***d1
2022-08-16T15:33:18.3391118Z CommandName                   : Az.ContainerInstance.internal\Invoke-AzContainerInstanceCommand
2022-08-16T15:33:18.3393542Z FullCommandName               : Invoke-AzContainerInstanceCommand_ExecuteExpanded
2022-08-16T15:33:18.3395650Z ParameterSetName              : __AllParameterSets
2022-08-16T15:33:18.3397954Z User-Agent                    : AzurePowershell/v7.5.***,PSVersion/v7.2.6,Az.ContainerInstance/***.1.***
2022-08-16T15:33:18.3398821Z 
2022-08-16T15:33:18.3400445Z Body:
2022-08-16T15:33:18.3401963Z {
2022-08-16T15:33:18.3403522Z   "terminalSize": {
2022-08-16T15:33:18.3405130Z     "rows": 4***,
2022-08-16T15:33:18.3406719Z     "cols": 8***
2022-08-16T15:33:18.3408267Z   },
2022-08-16T15:33:18.3409833Z   "command": "sleep 1***"
2022-08-16T15:33:18.3411621Z }
2022-08-16T15:33:18.3412122Z 
2022-08-16T15:33:18.3414137Z [***m
2022-08-16T15:33:18.3416176Z [***;1mDEBUG: BeforeCall: [***m
2022-08-16T15:33:19.0099705Z [***;1mDEBUG: ============================ HTTP RESPONSE ============================
2022-08-16T15:33:19.0102215Z 
2022-08-16T15:33:19.0103826Z Status Code:
2022-08-16T15:33:19.0105298Z OK
2022-08-16T15:33:19.0105742Z 
2022-08-16T15:33:19.0107278Z Headers:
2022-08-16T15:33:19.0109190Z Cache-Control                 : no-cache
2022-08-16T15:33:19.0111297Z Pragma                        : no-cache
2022-08-16T15:33:19.0114349Z Vary                          : Accept-Encoding
2022-08-16T15:33:19.0117724Z x-ms-request-id               : centralus:1fb61e91-5d81-42e7-8b8f-9d2a***2119177
2022-08-16T15:33:19.0120144Z x-ms-ratelimit-remaining-subscription-writes: 1199
2022-08-16T15:33:19.0123835Z x-ms-correlation-request-id   : ***422c***f-5b***a-4bba-af94-e8f115***48258
2022-08-16T15:33:19.0140772Z x-ms-routing-request-id       : NORTHCENTRALUS:2***22***816T15***19Z:***422c***f-5b***a-4bba-af94-e8f115***48258
2022-08-16T15:33:19.0143466Z Strict-Transport-Security     : max-age=***15***6***; includeSubDomains
2022-08-16T15:33:19.0145771Z X-Content-Type-Options        : nosniff
2022-08-16T15:33:19.0148218Z Date                          : Tue, 16 Aug 2***22 15:***:18 GMT
2022-08-16T15:33:19.0149047Z 
2022-08-16T15:33:19.0150646Z Body:
2022-08-16T15:33:19.0152167Z {
2022-08-16T15:33:19.0155500Z   "webSocketUri": "wss://sbzw1av9vdpqez1.centralus.atlas.cloudapp.azure.com:19***9***/execstream/sessionId/f855aed9-1247-42bb-856b-8d5c58917***99?api-version=1.***&rows=4***&cols=8***",
2022-08-16T15:33:19.0158643Z   "password": "***[[EOM]]"
2022-08-16T15:33:19.0160494Z }
2022-08-16T15:33:19.0160988Z 
2022-08-16T15:33:19.0163593Z [***m
2022-08-16T15:33:19.0166192Z [***;1mDEBUG: ResponseCreated: [***m
2022-08-16T15:33:19.0168491Z [***;1mDEBUG: BeforeResponseDispatch: [***m
2022-08-16T15:33:19.1994835Z [***;1mDEBUG: Finally: [***m
2022-08-16T15:33:19.2013044Z [***;1mDEBUG: CmdletAfterAPICall: [***m
2022-08-16T15:33:19.2016337Z [***;1mDEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process[***m
2022-08-16T15:33:19.2019043Z [***;1mDEBUG: CmdletProcessRecordAsyncEnd: [***m
2022-08-16T15:33:19.2027369Z [***;1mDEBUG: CmdletProcessRecordEnd: [***m
2022-08-16T15:33:19.2622963Z [***;1mDEBUG: AzureQoSEvent: Module: Az.ContainerInstance:***.1.***; CommandName: Invoke-AzContainerInstanceCommand; PSVersion: 7.2.6; IsSuccess: True; Duration: ***:***:***.9***7***985[***m```

Environment data

2022-08-16T15:33:19.3568995Z [***2;1mName                      Value[***m
2022-08-16T15:33:19.3571373Z [***2;1m----                      -----[***m
2022-08-16T15:33:19.3614794Z PSVersion                 7.2.6
2022-08-16T15:33:25.5016456Z PSVersion                 7.2.6
2022-08-16T15:33:25.5018308Z PSEdition                 Core
2022-08-16T15:33:25.5020047Z GitCommitId               7.2.6
2022-08-16T15:33:25.5022505Z OS                        Linux 5.15.***-1***17-azure #2***~2***.***4.1-Ubuntu SMP Fri A…
2022-08-16T15:33:25.5024481Z Platform                  Unix
2022-08-16T15:33:25.5029719Z PSCompatibleVersions      {1.***, 2.***, ***.***, 4.***…}
2022-08-16T15:33:25.5031735Z PSRemotingProtocolVersion 2.***
2022-08-16T15:33:25.5033467Z SerializationVersion      1.1.***.1
2022-08-16T15:33:25.5035172Z WSManStackVersion         ***.***

Module versions

This output is very garbled, but it should be Az.ContainerInstance 3.0.1 and Az.Accounts 2.9.1.

Error output

-08-16T15:33:25.3947000Z [***;1mDEBUG: 15:***:25 - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.[***m
2022-08-16T15:33:25.3949976Z [***;1mDEBUG: 15:***:25 - using account id '***'...[***m
2022-08-16T15:33:25.3953918Z [***;1mDEBUG: 15:***:25 - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].[***m
2022-08-16T15:33:25.3988184Z [***;1mWARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
2022-08-16T15:33:25.3991335Z The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.
2022-08-16T15:33:25.4000431Z Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.[***m
2022-08-16T15:33:25.4003421Z 
2022-08-16T15:33:25.4005766Z The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.
2022-08-16T15:33:25.4006810Z 
2022-08-16T15:33:25.4043893Z [***;1mDEBUG: 15:***:25 - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].[***m
2022-08-16T15:33:25.4047452Z [***;1mDEBUG: AzureQoSEvent: Module: Az.Accounts:2.9.1; CommandName: Resolve-AzError; PSVersion: 7.2.6; IsSuccess: True; Duration: ***:***:***.***59***98[***m
2022-08-16T15:33:25.4993409Z [***;1mDEBUG: Finish sending metric.[***m
2022-08-16T15:33:25.4997433Z [***;1mDEBUG: 15:***:25 - ResolveError end processing.[***m
dingmeng-xue commented 2 years ago

Thanks for reporting. Invoke-AzContainerInstanceCommand establishes standard output channel between PowerShell console can ContainerInstance. We have no idea what difference is in ADO. We are contacting engineering team.

lbouriez commented 8 months ago

@andysbolton , I know it's far but I am encountering the same issue, running Invoke-AzContainerInstanceCommand locally works but in my azure devops pipeline, it takes less than one second and i find that the command didn't properly run in the container. Did you by any chance find a solution ?

andysbolton commented 8 months ago

@lbouriez I haven't tested newer versions of azure-powershell and am not currently utilizing it, but when I was running into this issue initially, I worked around by manually making a post to the Containers endpoint and streaming back the results via the websocket address in the response. https://gist.github.com/andysbolton/88d70684e447fcb3a291f161d8631641

lbouriez commented 8 months ago

Thank you for your answer, I will have a look :)