[BUG]: AzureFileCopy@5 cleanup fails in GetJobResults with "failed to parse user input due to error: cannot use wildcards in the path section of the URL..."

[fowie]

New issue checklist

• [X] I searched for existing GitHub issues
• [X] I read pipeline troubleshooting guide
• [X] I checked how to collect logs

Task name

AzureFileCopy

Task version

5.234.0

Issue Description

The task began to fail today with the error message: failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A We only have a trailing slash in SourcePath in the command definition, but from my inspection, it looks like it's failing because the blob storage URL has 5 asterisks appended to the end. Those asterisks are not something I've input into the task, and they aren't part of the UUID, as you can see from earlier in the log

Environment type (Please select at least one enviroment where you face this issue)

• [X] Self-Hosted
• [ ] Microsoft Hosted
• [ ] VMSS Pool
• [ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

2024-02-08T20:33:01.1792738Z ##[debug][Azure Call]Creating container: 99c69625-ee4a-45f7-86a0-11bcfc92588b in storage account: 3dtmbinarystorage
2024-02-08T20:33:02.7981716Z ##[debug][Azure Call]Created container: 99c69625-ee4a-45f7-86a0-11bcfc92588b successfully in storage account: 3dtmbinarystorage
...
contact me directly @microsoft for full log if necessary
...
2024-02-08T20:33:28.4016542Z VERBOSE: Exporting function 'New-PSWorkflowSession'.
2024-02-08T20:33:28.4033884Z VERBOSE: Exporting alias 'nwsn'.
2024-02-08T20:33:29.0328279Z ##[debug]JobId: '2', JobState: 'running', ComputerName: '$(WSUSVMFQDN)'
2024-02-08T20:33:29.0352474Z ================================================ $(WSUSVMFQDN) ================================================
2024-02-08T20:33:29.1798996Z ##[command]& 'C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo -NoProfile -NonInteractive 
2024-02-08T20:33:29.1800960Z -ExecutionPolicy Unrestricted -Command ". 'C:\Users\3DTM\AppData\Local\Temp\564badbc-2273-449c-b27a-c207ee9d8467.ps1'"
2024-02-08T20:33:29.1801744Z 
2024-02-08T20:33:34.2738043Z ##[debug]JobId: '2', JobState: 'completed', ComputerName: '$(WSUSVMFQDN)'
2024-02-08T20:33:34.2762788Z ================================================ wsus-3dtm.uksouth.cloudapp.azure.com ================================================
2024-02-08T20:33:34.2825861Z VERBOSE: Using default AzCopy arguments for dowloading to VM
2024-02-08T20:33:34.2829285Z 
2024-02-08T20:33:34.3144160Z VERBOSE: ##[command] & azcopy copy 
2024-02-08T20:33:34.3147566Z 
2024-02-08T20:33:34.3635588Z "https://$(StorageResourceURL)/99c69625-ee4a-45f7-86a0-11bcfc92588b*****" 
2024-02-08T20:33:34.3638378Z 
2024-02-08T20:33:34.3925065Z "E:\MSUS\WsusContent\TRANSFER" --recursive --log-level=INFO
2024-02-08T20:33:34.3928494Z 
2024-02-08T20:33:34.4195491Z 
2024-02-08T20:33:34.4199248Z 
2024-02-08T20:33:34.4472492Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A
2024-02-08T20:33:34.4475185Z 
2024-02-08T20:33:34.4752474Z ##[debug][3DTM-Glasgow-WS]LASTEXITCODE is 1
2024-02-08T20:33:34.4754589Z 
2024-02-08T20:33:34.4902948Z ##[debug]Leaving Get-JobResults.
2024-02-08T20:33:34.4992830Z ##[debug]Leaving Run-RemoteScriptJobs.

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

steps:
- task: AzureFileCopy@5
  displayName: 'Copy 3DTM Launcher Service to WSUS'
  inputs:
    SourcePath: '"C:\Telemedicine3DOneDrive\OneDrive\FullSystemDeploy\$(Build.SourceBranchName)\3DTMLauncherService.*"'
    azureSubscription: 'Peabody Pipeline upload'
    Destination: AzureVMs
    storage: 3dtmbinarystorage
    resourceGroup: 3DTELEMEDECINE
    MachineNames: '$(WSUSVMName)'
    vmsAdminUserName: '$(WSUSAdminUsername)'
    vmsAdminPassword: '$(WSUSAdminPassword)'
    TargetPath: '$(WSUSDestinationFolder)'

[jiggybyte]

Same error...how are we expected to use the V5 task to remove dependencies on AzureRM if V5 is broken? Classic MS.

[jiggybyte]

I’m working around this by downloading the SXS version of this task…fixed the bad syntax in the power shell file…the azcopy command for copying from the storage account to the VM needed a question mark character between the container and the SAS params.

we shouldn’t have to work around these sorts of bugs all the time.

[TomasMalecek]

This affects AzureFileCopy@4, too. This happens when Azure PS 11.0.0+ is used. More specifically Az.Storage 6.0.0+. There was a breaking change in New-AzStorageContainerSASToken -- the return value is no longer prefixed with a question mark. The code of the tasks relies on that, though.

https://learn.microsoft.com/en-us/powershell/azure/release-notes-azureps#azstorage-600 https://learn.microsoft.com/en-us/powershell/azure/migrate-az-11.0.0#azstorage

The task log is not very helpful as it displays the source URI with five asterisks appended, while in reality, a slash, an asterisk, and the SAS token are appended.

The New-AzStorageContainerSASToken cmdlet is called in Generate-AzureStorageContainerSASToken defined in AzureUtilityAz1.0.ps1. That one is called from Copy-FilesToAzureVMsFromStorageContainer defined in Utility.ps1, which is called from AzureFileCopy.ps1 itself. There, it can be seen that Copy-FilesToAzureVMsFromStorageContainer gets the "fileCopyJobScript" scriptblock from AzureFileCopyRemoteJob.ps1. At the bottom, see the uses of containerSasToken: "$containerURL/*$containerSasToken"

[TomasMalecek]

The code of AzureFileCopy@3 differs a lot, but it is impacted the same way. Getting this error, which seems to be from AzCopy.exe itself:

[ERROR] The syntax of the command is incorrect. Invalid SAS token in parameter "SourceSAS".

[fowie]

I’m working around this by downloading the SXS version of this task…fixed the bad syntax in the power shell file…the azcopy command for copying from the storage account to the VM needed a question mark character between the container and the SAS params.

we shouldn’t have to work around these sorts of bugs all the time.

I've never done a custom SXS version, can you point me to some details on how to get this working?

[jiggybyte]

https://github.com/jessehouwing/azure-pipelines-tasks-zips/releases/tag/m236-tasks

You can download all of the packages here...the side by side (SXS) packages are there as well. I had to extract the "broken" file from their archive and then fix it and add it back in...then you can upload that file to your onprem TFS or Azure Devops and use it alongside the normal task.

There's a command line tool called tfx you likely have installed locally, you can login to that and use it to add/remove tasks to your onprem tfs. I think azure devops has a way to just do that in the web gui.

This powershell script was also handy: https://github.com/jessehouwing/azure-pipelines-tasks-zips/blob/main/scripts/install-task.ps1

[haf76348g78]

I’m working around this by downloading the SXS version of this task…fixed the bad syntax in the power shell file…the azcopy command for copying from the storage account to the VM needed a question mark character between the container and the SAS params.

Experiencing the same problem today, but was working yesterday and before. Same root cause (missing question mark). Don't understand how this could suddenly start happening today?

[marcusrichards]

Same error in FileCopy 4 and 5 - massive impact on our deployments. Urgent fix needed

[haf76348g78]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https://<your storage>/<foo>/*?$(<bar>.StorageContainerSasToken)" "<your destination>" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

[marcusrichards]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https:////*?$(.StorageContainerSasToken)" "" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

Thanks, couple of questions - is the output variable "bar" used anywhere - is that how the azcopy knows which storage container to copy? is the StorageContainerSasToken automatically set to the correct value?

[haf76348g78]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https:////*?$(.StorageContainerSasToken)" "" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

Thanks, couple of questions - is the output variable "bar" used anywhere - is that how the azcopy knows which storage container to copy? is the StorageContainerSasToken automatically set to the correct value?

Sorry, updated my original comment. I forgot to escape the angle brackets.

[volaru77]

Same here, started happening today with AzureFileCopy@4.

Looking at the logs, yesterday: ... Version : 4.236.1 ... Import-Module -Name C:\Modules\az_9.3.0\Az.Accounts\2.16.0\Az.Accounts.psd1 -Global

And today: ... Version : 4.237.1 ... Import-Module -Name C:\Modules\az_11.3.1\Az.Accounts\2.16.0\Az.Accounts.psd1 -Global

Is there a way to specify the version / module in the task parameters?

[marcusrichards]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https:////*?$(.StorageContainerSasToken)" "" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

Thanks, couple of questions - is the output variable "bar" used anywhere - is that how the azcopy knows which storage container to copy? is the StorageContainerSasToken automatically set to the correct value?

Sorry, updated my original comment. I forgot to escape the angle brackets.

This worked perfectly - thank you @haf76348g78 :) Do you have a quick way to install the WinRMCustomScriptExtension? I am using File Copy to do that still, then let it fail, then use your azcopy to pick up the pieces. But would be nice to do the WinRM install manually outside the File Copy.

[alundgren]

I’m working around this by downloading the SXS version of this task…fixed the bad syntax in the power shell file…the azcopy command for copying from the storage account to the VM needed a question mark character between the container and the SAS params.

we shouldn’t have to work around these sorts of bugs all the time.

If anyone else comes across this and struggles to find what to change like I did this is what we changed that seemed to work.

In Utility.ps1 there is a function function Copy-FilesToAzureVMsFromStorageContainer which has this:

$scriptBlockArgs = " -containerURL '$containerURL' -targetPath '$targetPath' -containerSasToken '$containerSasToken' -additionalArguments '$additionalArguments'"

That should be this instead:

$scriptBlockArgs = " -containerURL '$containerURL' -targetPath '$targetPath' -containerSasToken '?$containerSasToken' -additionalArguments '$additionalArguments'"

[Griffa87]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https:////*?$(.StorageContainerSasToken)" "" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

Thanks, couple of questions - is the output variable "bar" used anywhere - is that how the azcopy knows which storage container to copy? is the StorageContainerSasToken automatically set to the correct value?

Sorry, updated my original comment. I forgot to escape the angle brackets.

This worked perfectly - thank you @haf76348g78 :) Do you have a quick way to install the WinRMCustomScriptExtension? I am using File Copy to do that still, then let it fail, then use your azcopy to pick up the pieces. But would be nice to do the WinRM install manually outside the File Copy.

Can confirm this works. Thank you so much! saved me a lot of headache this morning.

[gusAntonis]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https:////*?$(.StorageContainerSasToken)" "" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

Thanks, couple of questions - is the output variable "bar" used anywhere - is that how the azcopy knows which storage container to copy? is the StorageContainerSasToken automatically set to the correct value?

Sorry, updated my original comment. I forgot to escape the angle brackets.

This worked perfectly - thank you @haf76348g78 :) Do you have a quick way to install the WinRMCustomScriptExtension? I am using File Copy to do that still, then let it fail, then use your azcopy to pick up the pieces. But would be nice to do the WinRM install manually outside the File Copy.

Can confirm this works. Thank you so much! saved me a lot of headache this morning.

Can you guys provide some more detailed reference on the changes applicable, Little confused on the usage of the output variables and the powershell task @Griffa87 @marcusrichards

[gusAntonis]

If it helps, I worked around it like this:

1. Change the Destination Type of the Azure File Copy task to Azure Blob
2. Put something in Container Name like "foo"
3. Check the Clean Target checkbox
4. Under Output Variables, put something in Reference Name like "bar"
5. Add a PowerShell on Target Machines task
6. Enter the server name of your target machine and login credentials
7. Add the following as an Inline Script: azcopy copy "https://<your storage>/<foo>/*?$(<bar>.StorageContainerSasToken)" "<your destination>" --recursive --log-level=INFO

The reconfigured Azure File Copy task will do the "copy to blob" bit, and the new task will do the "copy from blob to target" bit.

@haf76348g78 i am still facing issues with the powershell task used for copying the file from blob to the VM, can you share the reference for the tasks to be modified, a sample yaml would really help a lot

[AndrewBates666]

We have implemented the workaround and this was working last week but has started failing this week. I'm in the process of investigating the problem.

There is obviously a bug that has been introduced in the AzureFileCopy task with the missing ? between the url and access token; can this please be fixed as a priority as it is impacting our releases?!?

[gusAntonis]

We have implemented the workaround and this was working last week but has started failing this week. I'm in the process of investigating the problem.

There is obviously a bug that has been introduced in the AzureFileCopy task with the missing ? between the url and access token; can this please be fixed as a priority as it is impacting our releases?!?

So the copy to blob and then copy to VM started failing for you as well.

[AndrewBates666]

Yes, the copy to the blob is working but the copying to the VM has started to fail this week. As I say looking into it now to understand why. We ideally need the bug fixing though as this isn't the first time AzureFileCopy has been broken.

[gusAntonis]

Yes, the copy to the blob is working but the copying to the VM has started to fail this week. As I say looking into it now to understand why. We ideally need the bug fixing though as this isn't the first time AzureFileCopy has been broken.

So i also implemented the solution today, we are also able to copy to the blob but when we try to copy to the VM, We don't get any errors, rather just the transfer fails, we are using the az-copy for copying the files from blob to VM

[AndrewBates666]

Seems that azcopy.exe isn't available on the Azure Hosted Build agents which is causing the copy from blob to the VM to fail... We're using the windows-2022 vmImage.

azcopy : The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or...

[volaru77]

Copying from the blob to the VM is still working as of now if azcopy is invoked from the VM using the PowerShellOnTargetMachines@3 task

[gusAntonis]

Copying from the blob to the VM is still working as of now if azcopy is invoked from the VM using the PowerShellOnTargetMachines@3 task

@volaru77 can you share a reference of your task with PowerShellOnTargetMachines@3 task Would like to have a look. Thanks

[volaru77]

Copying from the blob to the VM is still working as of now if azcopy is invoked from the VM using the PowerShellOnTargetMachines@3 task

@volaru77 can you share a reference of your task with PowerShellOnTargetMachines@3 task Would like to have a look. Thanks

Assuming the task to copy the files to the blob is named 'blobcopy' and the blob container name is 'container' it would look like this:

- task: PowerShellOnTargetMachines@3
  inputs:
    Machines: 'server.region.cloudapp.azure.com'
    UserName: 'server'
    UserPassword: '$(password)'
    InlineScript: |
      azcopy copy "https://storageaccount.blob.core.windows.net/container/*?$(blobcopy.StorageContainerSasToken)" "C:\Folder" --recursive --log-level=INFO

[gusAntonis]

@volaru77 can you share the modified azurefilecopytask as well i just want to reference the Output Variables part, facing some issues in that, It would be a great help.

[marcusrichards]

@volaru77 can you share the modified azurefilecopytask as well i just want to reference the Output Variables part, facing some issues in that, It would be a great help.

Blobcopy here is the name of the blob copy task, that task creates the property of the token which is populated into the task name variable. Hope this helps

[MouhcineKarbach]

@volaru77 it's like this

      - task: AzureFileCopy@5
        displayName: 'Uploadfiles to AzureBlob'
        name : 'blobcopy'
        inputs:
          SourcePath: '$(Build.SourcesDirectory)'
          azureSubscription: 'your subscripton name'
          Destination: 'AzureBlob'
          storage: 'yourstorageaccountName'
          ContainerName: 'buildcontainer'

      - task: PowerShellOnTargetMachines@3
        displayName: 'Donwload files on rempte machine'
        inputs:
          Machines: 'machineIP'
          UserName: '$(VMAdminUsername)'
          UserPassword: '$(VMAdminPassword)'
          InlineScript: |
            azcopy copy "https://yourstorageaccountName.blob.core.windows.net/buildcontainer/?$(blobcopy.StorageContainerSasToken)" "C:\wwwroot\ApplicationFolder" --recursive --log-level=DEBUG

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[TomasMalecek]

I believe this still needs fixing.

[TomasMalecek]

By the way, I worked around the issue by using PS Remoting and Copy-Item. As a side effect, the copying became much faster as the copying is direct, not via a storage account.

[marcusrichards]

It appears this was a breaking change that will not be reverted, the ? had been dropped from the tokens returned from the file copy task. We now have a script to check if the ? is missing and manually add it back to the query string before appending to the url. Very poor practice of Microsoft to introduce a regression change like this.

[marcusrichards]

By the way, I worked around the issue by using PS Remoting and Copy-Item. As a side effect, the copying became much faster as the copying is direct, not via a storage account.

Can you share your code for this? We had exactly the same problem, main pain being the WinRM script which is very difficult to recreate if the file copy task is broken.

[TomasMalecek]

Yes, the script was not trivial to create, @marcusrichards. I created it once, stored it in a Git repo and reused it by letting the pipeline pull its latest version to the agent. Sadly, I created it as part of a contract work and I cannot share it outside the organization.

But I believe we could use WindowsMachineFileCopy@2 from the mmajcica.win-rm-file-copy extension instead. We already use it in a different pipeline and I am looking forward to trying to replace the custom script with it. It should work the same way while being self-contained. Not sure but maybe the script would be able to run even on Linux with PowerShell Core, while WindowsMachineFileCopy@2 will not because it is implemented in PowerShell and such tasks work only on Windows. It does not matter to us, though, as we need a Windows agent anyway.

• WindowsMachineFileCopy@2 @ MS Learn
• WinRm File Copy @ VS Marketplace
• mmajcica/win-rm-file-copy @ GitHub

[marcusrichards]

Thank you! AzureFileCopy task is one of the flakiest we use, several times in the last couple of years it has been broken, but we need it so try to work around these issues!