yonzhan
commented
11 months ago Thank you for opening this issue, we will look into it.
Open DanteMustCode opened 11 months ago
yonzhan
commented
11 months ago Thank you for opening this issue, we will look into it.
ghost
commented
11 months ago Thank you for your feedback. This has been routed to the support team for assistance.
bebound
commented
11 months ago This warn info is expected as we bump cryptography to latest version in 2.51.
You can try to install the 64-bit package and migrate to it. see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli#migrate-to-64-bit-azure-cli. We will work with ADO team to migrate to 64-bit Azure CLI.
Alternatively, set failOnStderr to false to keep the pipeline running. The correct way to detect if a CLI command fails is to check the exit code
DanteMustCode
commented
11 months ago @bebound Thank you very much. I will close this issue as completed.
HenrikSommer-Energinet
commented
11 months ago If that is expected then please update your documentations also. There are no info about it here: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli You can't expected customer to find a pr branch.
HenrikSommer-Energinet
commented
11 months ago @DanteMustCode can you reopen it until your documentation is updated?
dylanhouston7
commented
11 months ago @DanteMustCode can you reopen it until your documentation is updated?
I agree. I work at Fortune 200 who is a client of Microsoft and this has impeded our ability to run any of our Azure Pipelines this morning, when they were working before. I need some documentation on how to best get around this issue.
I also don't think this should be logged as an exception since the message has the word "UserWarning" in it.
sugangin
commented
11 months ago I agree too. This may be a valid warning. But since it is being written as error, and most of the azure pipeline az cli tasks use "Fail on Standard Error" flag, it results in failure of the azure pipelines. This flag cannot be set to false as it would lead to the "actual/genuine script errors" to be ignored.
It will be hugely appreciated if this behavior can be fixed or provide Azure pipeline tasks the ability to chose older versions of Az cli.
sugangin
commented
11 months ago Also I would like to bring to your kind attention the magnitude of this issue is having on the customers. All of the scheduled Production deployments for the Customers who has chosen "Azure Cloud" and running their Azure pipelines on Microsoft Hosted agents, and using Azure CLI tasks for many deployment scripts are unable to move forward with the scheduled UAT and Production deployments.
dylanhouston7
commented
11 months ago Also I would like to bring to your kind attention the magnitude of this issue is having on the customers. All of the scheduled Production deployments for the Customers who has chosen "Azure Cloud" and running their Azure pipelines on Microsoft Hosted agents, and using Azure CLI tasks for many deployment scripts are unable to move forward with the scheduled UAT and Production deployments.
Absolutely agree 100% with this. This is impeding our ability to move code past the PR stage. Fwiw I've filed a Sev B Azure ticket on behalf of the company I work for, and linked to this Github issue within the ticket.
navba-MSFT
commented
11 months ago @sugangin @dylanhouston7 I am reopening this issue and checking this further internally.
navba-MSFT
commented
11 months ago @sugangin @dylanhouston7 Could you please follow the suggestion and workaround mentioned here.
jiasli
commented
11 months ago This flag cannot be set to false as it would lead to the "actual/genuine script errors" to be ignored.
@sugangin, Azure CLI never uses stderr stream as an error indicator. Any command can print to stderr without prior notice. Please see https://github.com/Azure/azure-cli/issues/18372 for more details regarding using failOnStderr with Azure CLI.
dylanhouston7
commented
11 months ago @sugangin @dylanhouston7 Could you please follow the suggestion and workaround mentioned here.
@navba-MSFT thanks for reopening the issue. Changing the "failOnStandardError" flag to false does keep our pipelines going and does unblock us. However, as @sugangin pointed out, doing this also ignores "actual/genuine" errors. This actually happened to me yesterday, after we implemented the workaround. When we were trying to deploy some keyvault reference changes with a functions app, we found that the slot failed to start. We found out that we didn't have access policies setup for the app to access the keyvault secrets.
Usually, this step
az deployment group what-if -g "$ResourceGroupName"
would have caught that error before we deployed anything to Azure. But because we did the workaround, we had to go to the deployed Functions app in Azure to see that the deploy slot failed to deploy because of that-- and we were able to view the error there in the Azure portal, but it is always much better to catch errors like this well before anything is deployed to Azure.
My Sev B Azure ticket is being looked at by Microsoft as well. 2308080040010326
Implemented workaround attached. azure-workaround.txt
nthompson4work
commented
11 months ago Our workaround was to implement what was described here.
At first we just pinned the cli version, azure-cli==2.51, but that then led to issue #27131.
We ended up adding the following which seems to have done the trick:
steps:
...
# Specify python version
- task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
architecture: 'x64'
# Update to latest Azure CLI version
- bash: pip install azure-cli==2.51 azure-core==1.28.0 --extra-index-url https://azurecliprod.blob.core.windows.net/edge
displayName: 'Upgrade Azure CLI'
...The downside to this however is two fold:
1) Need to run it at the beginning of every agent provisioning so any azure CLI tasks via scripts or task: AzureCLI@2 get the update. So if you provision multiple times in a pipeline, it needs to be done for each.
2) It is SLOW. We are seeing an average of 3 to 7 minutes for this mitigation step alone to install the 64bit version. in areas where this needs to be done multiple times, the pipeline time has now gone way up.
dylanhouston7
commented
11 months ago Our workaround was to implement what was described here.
At first we just pinned the cli version,
azure-cli==2.51, but that then led to issue #27131.We ended up adding the following which seems to have done the trick:
steps: ... # Specify python version - task: UsePythonVersion@0 inputs: versionSpec: '3.x' architecture: 'x64' # Update to latest Azure CLI version - bash: pip install azure-cli==2.51 azure-core==1.28.0 --extra-index-url https://azurecliprod.blob.core.windows.net/edge displayName: 'Upgrade Azure CLI' ...The downside to this however is two fold:
1. Need to run it at the beginning of every agent provisioning so any azure CLI tasks via scripts or `task: AzureCLI@2` get the update. So if you provision multiple times in a pipeline, it needs to be done for each. 2. It is _SLOW_. We are seeing an average of 3 to 7 minutes for this mitigation step alone to install the 64bit version. in areas where this needs to be done multiple times, the pipeline time has now gone way up.
Thanks for posting this. The first tier MS rep to get back to me suggested a similar workaround, which added 6-7 minutes to our pipeline run time.... which is unacceptable and also SLOW. Our PRs builds are what leverage this command so it is run all the time.
This is the workaround that he suggested. Just posting this in case someone is really stuck and needs to release code.
jobs:
- job:
dependsOn: ${{ parameters.DependsOn }}
displayName: "${{ parameters.ArmTemplateSubFolder }}/${{ parameters.Environment }}/${{ parameters.Region }}"
steps:
- checkout: none
- download: current
artifact: ${{ parameters.TemplateFolderPath }}
- bash: pip install -Iv azure-cli==2.50.0 --extra-index-url https://azurecliprod.blob.core.windows.net/edge
displayName: 'Use Specific Azure CLI' # 8/10/23 - Using a specific version of the CLI until this issue is resolved https://github.com/Azure/azure-cli/issues/27115
- task: AzureCLI@2 # this step is required to run bicep deploys that have a module reference to a private repository
displayName: "Testing Files"
inputs:
azureSubscription: ${{ parameters.AzureSubscription }}
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
cutting this out for brevity purposes
azurePowerShellVersion: latestVersion
errorActionPreference: 'stop' # Optional. Options: stop, continue, silentlyContinue
failOnStandardError: true # Optional
pwsh: true
cutecycle
commented
11 months ago Our workaround was to implement what was described here.
At first we just pinned the cli version,
azure-cli==2.51, but that then led to issue #27131.We ended up adding the following which seems to have done the trick:
steps: ... # Specify python version - task: UsePythonVersion@0 inputs: versionSpec: '3.x' architecture: 'x64' # Update to latest Azure CLI version - bash: pip install azure-cli==2.51 azure-core==1.28.0 --extra-index-url https://azurecliprod.blob.core.windows.net/edge displayName: 'Upgrade Azure CLI' ...The downside to this however is two fold:
- Need to run it at the beginning of every agent provisioning so any azure CLI tasks via scripts or
task: AzureCLI@2get the update. So if you provision multiple times in a pipeline, it needs to be done for each.- It is SLOW. We are seeing an average of 3 to 7 minutes for this mitigation step alone to install the 64bit version. in areas where this needs to be done multiple times, the pipeline time has now gone way up.
do you have to pip install the specific azure cli version? python x64 seems to already be installed on Microsoft-hosted agents, and just UsePythonVersion could be enough in that context (I'm not sure if you're on self-hosted agents).
cutecycle
commented
11 months ago Our workaround was to implement what was described here. At first we just pinned the cli version,
azure-cli==2.51, but that then led to issue #27131. We ended up adding the following which seems to have done the trick:steps: ... # Specify python version - task: UsePythonVersion@0 inputs: versionSpec: '3.x' architecture: 'x64' # Update to latest Azure CLI version - bash: pip install azure-cli==2.51 azure-core==1.28.0 --extra-index-url https://azurecliprod.blob.core.windows.net/edge displayName: 'Upgrade Azure CLI' ...The downside to this however is two fold:
- Need to run it at the beginning of every agent provisioning so any azure CLI tasks via scripts or
task: AzureCLI@2get the update. So if you provision multiple times in a pipeline, it needs to be done for each.- It is SLOW. We are seeing an average of 3 to 7 minutes for this mitigation step alone to install the 64bit version. in areas where this needs to be done multiple times, the pipeline time has now gone way up.
do you have to
pip installthe specific azure cli version? python x64 seems to already be installed on Microsoft-hosted agents, and just UsePythonVersion could be enough in that context (I'm not sure if you're on self-hosted agents).
answering myself:
the second line is necessary: ensures the openssl dependency is 64 bit upon reinstallation
jiasli
commented
10 months ago Installing with pip such as pip install azure-cli==2.51 is not officially supported.
@dylanhouston7, using failOnStandardError to detect if az deployment group what-if -g "$ResourceGroupName" succeeds is an incorrect usage. See https://github.com/Azure/azure-cli/issues/18372#issuecomment-1671014024 for how to handle the exit code of a native command correctly.
MattJeanes
commented
10 months ago We have worked around this by setting failOnStdErr: false (or removing it which is the default) in our Azure Pipelines and updating the PowerShell scripts to check $LASTEXITCODE after each call to a native command like the Azure CLI:
az <insert command here>
if ($LASTEXITCODE -ne 0) {
Write-Error "az exited with code $LASTEXITCODE"
}Note that PowerShell 7.3 includes an experimental feature to do this for you: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_preference_variables?view=powershell-7.3#psnativecommanduseerroractionpreference
Bash scripts also can do this with set -e at the top of the script
cutecycle
commented
10 months ago @MattJeanes @jiasli But then, isn't part of the problem here that the UserWarning is changing the exitCode of this native call to 1?
jiasli
commented
10 months ago @MattJeanes's solution is correct. This is exactly what I provided in https://github.com/Azure/azure-cli/issues/18372#issuecomment-1671014024.
the UserWarning is changing the exitCode of this native call to 1
UserWarning is merely a message printed to stderr. It never changes the exit code. This can be verified with
> C:\Users\xxx\AppData\Local\Programs\Python\Python310-32\python.exe -c "import cryptography.hazmat.bindings.openssl.binding"
<string>:1: UserWarning: You are using cryptography on a 32-bit Python on a 64-bit Windows Operating System. Cryptography will be significantly faster if you switch to using a 64-bit Python.
> echo $LASTEXITCODE
0
bowlerma
commented
10 months ago We're also now started seeing this problem with Microsoft's hosted agents which was causing an otherwise working task to suddenly start failing.
If it's a warning, why is it being written to stderr?
Where's the notification that this breaking change was being introduced, and why have the agents been updated to this version if they're not compatible with it?
MattJeanes
commented
10 months ago Standard error (stderr) output is also often used for warnings and even sometimes informational messages which is why I never recommend using failOnStdErr as it will cause problems like this. The only reliable metric for task success / failure is the exit code
dylanhouston7
commented
10 months ago Regarding the Azure ticket I opened on behalf of the company I work for, a rep got back to me and told me to try my pipeline without the workaround of using a specific Azure CLI version. I did, and it worked today. Anyone else seeing successful results?
not-not-kevin
commented
10 months ago Regarding the Azure ticket I opened on behalf of the company I work for, a rep got back to me and told me to try my pipeline without the workaround of using a specific Azure CLI version. I did, and it worked today. Anyone else seeing successful results?
Confirmed. I re-ran my pipelines and they work successfully (without the "workaround'). However, moving forwards I will continue to check the last exit code vs failing on standard error.
bebound
commented
10 months ago Runner-image is using 64-bit CLI by default in https://github.com/actions/runner-images/pull/8096 I think this issue is solved for pipeline users.
bowlerma
commented
10 months ago Yep, working here too now. As per above, 64-bit Python seems to now be used on the Microsoft hosted agents, so the warning goes away.
dylanhouston7
commented
10 months ago Thanks for verifying all. When I removed the workaround for various other repos that I work with, they all worked this morning as well.
I did not get much detail on what the fix was, other than "...it is a full implementation for the MS hosted agents on the latest version. Your other repos should be fine as long as they run with the latest version of the Microsoft Hosted Agent."
I'm going to have them close the Azure ticket now.
dylanhouston7
commented
10 months ago One last note, I got these details once my ticket was closed:
• Issue: Azure CLI with version 2.51.0 having a cryptography error on a 32-bit Python on Windows OS on 64-bit version.
• Root Cause: The Azure CLI version 2.51.0 used on Microsoft Hosted Agents was working on the 32 bits version instead of 64 bits version
• Actions taken for the resolution: For the resolution of the case, I opened an internal report sharing the github thread: [Azure CLI 2.51.0 (August 01, 2023) introduces Python UserWarning of cryptography for extensions · Issue #27115 · Azure/azure-cli · GitHub](https://github.com/Azure/azure-cli/issues/27115) and the investigation, workarounds taken for this request.
o Final resolution: A fix implementation based on the error was made for the latest version of Microsoft Hosted Agents by switching the version of Azure CLI to the 64 bits and mitigate the current behavior mentioned before.
martin-bmc
commented
10 months ago The installation documentation does not show how to install the 64 bit version using winget (and I can not see such a package available). Is this in progress and if so any ETA?
bebound
commented
10 months ago @martin-bmc x64 version for winget is not in the roadmap yet. We can't replace the download link to 64-bit as there are some manual steps to migrate to 64bit. I think we need to keep both 32-bit and 64-bit and add some instructions after installation.
HenrikSommer-Energinet
commented
10 months ago @martin-bmc x64 version for winget is not in the roadmap yet. We can't replace the download link to 64-bit as there are some manual steps to migrate to 64bit. I think we need to keep both 32-bit and 64-bit and add some instructions after installation.
efd7887
commented
1 month ago So you are saying that the right hand doesnt know what the left hand is doing over there at Microsoft? What is going on over there? Is this a feature? This is like selling a car with 2 wheels. Constant issues with VS Code. Is having a stable command line interface other than DOS on Microsoft's roadmap because I have to tell you, if I got paid to research and fix all of the issues with VS Code and modules in PowerShell and CLI I may not have to work. Effectively we are doing our jobs and your jobs.
This is my opinion only and does not represent Microsoft in any way
Open Source Community Tools = We arent sure we know what we are doing so we will rely on the intelligent customers to fix stuff for us.
If I architected and built solutions that only half worked and then someone had to come behind my, find the issue, diagnose and fix I would be fired. Just saying......
tssv2896
commented
1 month ago Runner-image is using 64-bit CLI by default in actions/runner-images#8096 I think this issue is solved for pipeline users.
Worked perfectly !
Describe the bug
When Azure CLI 2.51.0 (August 01, 2023) is installed,
arcapplianceextension produces a PythonUserWarning:This leads to PowerShell
RemoteExceptionwhich fails Azure Pipelines.We have tried to uninstall 2.51.0 and install 2.50.0, and the error disappears.
Related command
Errors
Issue script & Debug output
Expected behavior
No error as 2.50.0.
Environment Summary
Additional context
https://azcliprod.blob.core.windows.net/msi/azure-cli-2.51.0.msi https://azcliprod.blob.core.windows.net/msi/azure-cli-2.50.0.msi