Azure File Copy build task not available on hosted Ubuntu build agent

[benne]

Environment

• Server - Azure Pipelines Account name: chaosbyte Team project name: D Test Build definition name: D Test-CI Build number: 9

• Agent - Hosted: Queue name: Not sure It is hosted a Ubuntu 16.04 build agent

Issue Description

I got a project building some pulling from an external Git repository, and building it with gcc through a simple bash script - no problem. However, when I attempt to use the Azure File Copy task to copy over the single output file to a Azure Blob Storage container, I am told that this task won't run on Linux. Is there any plans to make Azure File Copy work on the hosted Ubuntu 16.04 build agent, or do you have any other suggestion on how to achieve the same result with existing tasks that will run on this hosted build server?

Error logs

2018-11-27T18:03:41.7033273Z ##[section]Starting: AzureBlob File Copy
2018-11-27T18:03:41.7038323Z ==============================================================================
2018-11-27T18:03:41.7038468Z Task         : Azure File Copy
2018-11-27T18:03:41.7038567Z Description  : Copy files to Azure blob or VM(s)
2018-11-27T18:03:41.7038647Z Version      : 2.0.15
2018-11-27T18:03:41.7038738Z Author       : Microsoft Corporation
2018-11-27T18:03:41.7038823Z Help         : [More Information](https://aka.ms/azurefilecopyreadme)
2018-11-27T18:03:41.7038939Z ==============================================================================
2018-11-27T18:03:41.7244544Z ##[error]The current operating system is not capable of running this task. That typically means the task was written for Windows only. For example, written for Windows Desktop PowerShell.
2018-11-27T18:03:41.7258675Z ##[section]Finishing: AzureBlob File Copy

[chrispat]

@benne you could use the az cli to do the file copy and that task will work on Linux

[benne]

@chrisrpatterson az cli might be my best option for now, using a non-interactive service principal. I am gonna experiment a bit with this.

Still interested in if this is gonna make it to the road-map at some point. It would be nice to use the build-in build task for future projects :)

[chrispat]

@benne I will leave the roadmap for that task up to the team that owns the task. The az cli task should work with the same service principal you would have setup for the Azure File Copy task.

[benne]

@chrisrpatterson I went with a service principal with specific permissions to contribute to a specific storage account. Combined with an Azure CLI build task I was able to upload the assets to the blob storage. Success!

[kmkumaran]

@benne - Making Azure File copy xPlat is in our backlog. Adding @RoopeshNair . Until then using Azure CLI will be the right way to solve.

[RoopeshNair]

Bouncing an idea here, we are evaluating whether we can expose Azure storage as a file drive. When available you can copy the files over using SSH or Remote PS.

[luyseyal]

azcopy is available on Linux https://aka.ms/downloadazcopy-v10-linux

[mounarshisantra]

if we are using the az cli to achieve the same functionality, we are not getting the same flexibility as the Azure File Copy task in terms of getting the storage uri and sastoken as output. We need to make additional cli calls to achieve the same . So, it would be great if the file copy task works xPlat.

[motowilliams]

The AzureFileCopy task and the az cli workaround is not at feature parity with each other. I wonder how many teams, like I just did, spent time adding the task to the pipeline, watching it fail and then searching the internet for a couple of hours. This is how I found this issue. The docs on this are also weak.

[titimoby]

Is there any progress to this? It is very strange that tasks like JavaToolInstaller are able to copy installer file from Azure Storage but we can't use AzureFileCopy to copy files to Azure Storage.

[minhtrieu-aemo]

Trying to migrate our pipelines to use self-hosted Linux agent and discovered that its failing due to AzureFileCopy not being supported on Linux. It would be great if this can be made to work with Linux!

[weznagwama]

Another bump for an update on this.

[alexangas]

I've written a YAML task template for Azure Pipelines that fills this gap for now. You can find it on my blog here: https://www.alexangas.com/blog/missing-azurefilecopy-azcopy-task-solved/

Feedback is very welcome! Please raise as an issue on the repo for my blog rather than polluting this thread, thanks.

[veton]

For the time being, using windows image as a workaround:

  - deployment: deploy_staging
    environment: staging
    pool:
      vmImage: windows-latest
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: $(artifactName)

          - task: AzureFileCopy@3
            inputs:
              sourcePath: $(Pipeline.Workspace)/$(artifactName)
              azureSubscription: $(azureSubscription)
              destination: azureBlob
              storage: memes
              containerName: $web

[argium]

This also left me very confused. This obviously should work on linux hosts.

[tckb]

More than a year later, the docs are still stale. Thanks for the work around @veton

[jamiehaywood]

is there an update on this?

[jamiehaywood]

I wanted to share my own solution to this that works with vmImage:ubuntu-latest:

  - task: AzureCLI@1
    displayName: Az File Copy to Storage
    inputs:
      azureSubscription: $(azureSubscription)
      scriptLocation: inlineScript
      inlineScript: |
        az storage blob upload-batch \
          --destination \$web \
          --account-name "$(storageAccountName)" \
          --source "$(Agent.BuildDirectory)/$(outputDir)"

[sebastianrogers]

Critical thing here is that Azure File Copy will return a SAS Token allowing you to then access these files via ARM deployment.

[trondkla]

Any progress? This should be a thing for linux too :)

[xortim]

Not to +1 and "me too" but if the guidance from MS is to use azcopy, then they should support it in their products. What happened to MS ❤️ Linux? :trollface:

[beauof]

Thanks for sharing your workarounds! Yet, I am still waiting for the AzureFileCopy task on Linux agents because I can't always remember this issue until I see another failed pipeline. :)

[jwbrindle]

umm, have been hoping to see some kind of update on this for over a year, what gives guys, can we get some support on this? GA opened since azcopy v10 for linux, why couldnt this be integrated?

[cee-money]

+1, it would be great to see an update on this

[ascott18]

This is a terrible experience for your customers. Why is something so basic as copying files not supported on Linux agents the same as it is supported on Windows?

[nadesu]

@vijayma pls triage

[dancooktoptal]

+1

[ThePiz]

+1

[buggsdummy]

+1

[khenriksson]

+1

[boylec]

+1

[pstiegele]

+1

[crgarcia12]

+1

[FlorianJDF]

+1

[hacktick]

+1 (i can already see the headlines in the washington post newspaper in the year 2038 - "Microsoft has finalized file copy for linux pipelines - it is now in the testing stage and soon ready for RC stage") :-)

[benne]

Switched to Windows hosts for the specific project, but would still love to see a cross-platform solution that just works as intented :)

[alswinehartMRI]

@jamiehaywood thank you so much! Does anyone know how I can narrow down copying to a repo's subfolder? I'm trying this now with wild guesses and it's either erroring out or copying all the unnecessary files, which ends up taking over 60 minutes and erroring out anyway.

[johnynfulleffect]

+1, please allow to run on linux

[acgar]

+1

[snowjobgit]

+1

[SammySamer]

+1

[Jaykul]

Any chance this is finally being addressed in v5? 🙄

[hcoona]

Years & years later :(

[ngcdev]

I wonder if the "roadmap" is that kind of map of "where do you see yourself in 10 years"... a bunch of tasks work flawlessly in Linux hosts, but this one gets no attention

[dct2012]

+1

[michal-pawlowski]

+1

[gyermakavets]

+1

[Shaharajshaik]

+1

[n-alex-white]

Four and a half years (including a whole pandemic) and this still isn't working? Is it still on the backlog @kmkumaran ?

[ngcdev]

We changed our pipelines to use PowerShell (since now PowerShell core works in both OSes) and used az commands to do the copy instead. Working good so far.