search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.61k forks source link

[BUG]: AzureFileCopy not respecting AZCOPY_CONTENT_TYPE_MAP #18001

Open celluj34 opened 1 year ago

celluj34 commented 1 year ago

Task name

AzureFileCopy

Task version

5.218.0

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 10

Task log

2023-03-23T20:56:34.7027482Z ##[section]Starting: Copy addin files to storage account
2023-03-23T20:56:34.7350253Z ==============================================================================
2023-03-23T20:56:34.7350557Z Task         : Azure file copy
2023-03-23T20:56:34.7350776Z Description  : Copy files to Azure Blob Storage or virtual machines
2023-03-23T20:56:34.7351004Z Version      : 5.218.0
2023-03-23T20:56:34.7351173Z Author       : Microsoft Corporation
2023-03-23T20:56:34.7351433Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2023-03-23T20:56:34.7351769Z ==============================================================================
2023-03-23T20:56:37.7774545Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\2.10.2\Az.Accounts.psd1 -Global
2023-03-23T20:56:44.9720572Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2023-03-23T20:56:45.4329122Z ##[command]Clear-AzContext -Scope Process
2023-03-23T20:56:45.5208117Z ##[command]Connect-AzAccount System.Collections.Hashtable
2023-03-23T20:56:45.6167993Z ##[warning]The provided service principal secret will be included in the 'AzureRmContext.json' file found in the user profile ( C:\Users\***\.Azure ). Please ensure that this directory has appropriate protections.
2023-03-23T20:56:47.4516799Z 
2023-03-23T20:56:47.5352860Z ##[command]Set-AzContext System.Collections.Hashtable
2023-03-23T20:56:48.4188803Z Account                              SubscriptionName TenantId                             Environment
2023-03-23T20:56:48.4191681Z -------                              ---------------- --------                             -----------
2023-03-23T20:56:48.4192610Z *** TINC.CORP.DEV    *** AzureCloud 
2023-03-23T20:56:48.4193259Z 
2023-03-23T20:56:48.4193966Z Name               : TINC.CORP.DEV (***) - *** - 
2023-03-23T20:56:48.4195201Z                      ***
2023-03-23T20:56:48.4195624Z Account            : ***
2023-03-23T20:56:48.4196117Z Environment        : AzureCloud
2023-03-23T20:56:48.4196527Z Subscription       : ***
2023-03-23T20:56:48.4196964Z Tenant             : ***
2023-03-23T20:56:48.4197336Z TokenCache         : 
2023-03-23T20:56:48.4197650Z VersionProfile     : 
2023-03-23T20:56:48.4198069Z ExtendedProperties : {}
2023-03-23T20:56:48.4198278Z 
2023-03-23T20:56:49.2420386Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Resources\6.3.1\Az.Resources.psd1 -Global
2023-03-23T20:56:55.7612623Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Storage\5.0.0\Az.Storage.psd1 -Global
2023-03-23T20:56:59.5280273Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Compute\5.0.0\Az.Compute.psd1 -Global
2023-03-23T20:57:04.4797280Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Network\5.0.0\Az.Network.psd1 -Global
2023-03-23T20:57:05.6362372Z ##[warning]The names of some imported commands from the module 'Microsoft.Azure.PowerShell.Cmdlets.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-03-23T20:57:05.9638098Z ##[warning]The names of some imported commands from the module 'Az.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-03-23T20:57:09.8587393Z  mime: D:\agent-2\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.218.0\MimeMapping.json
2023-03-23T20:57:09.8631825Z ##[command] & "AzCopy\AzCopy.exe" login --service-principal --application-id "***" --tenant-id="***" --aad-endpoint "https://login.windows.net/"
2023-03-23T20:57:10.6634498Z INFO: If you set an environment variable by using the command line, that variable will be readable in your command line history. Consider clearing variables that contain credentials from your command line history.  To keep variables from appearing in your history, you can use a script to prompt the user for their credentials, and to set the environment variable.
2023-03-23T20:57:10.9402227Z INFO: AzCopy.exe: A newer version 10.17.0 is available to download
2023-03-23T20:57:10.9402531Z 
2023-03-23T20:57:11.0251544Z INFO: SPN Auth via secret succeeded.
2023-03-23T20:57:11.2663598Z INFO: AzCopy.exe: A newer version 10.17.0 is available to download
2023-03-23T20:57:11.2664030Z 
2023-03-23T20:57:11.2751824Z Uploading files from source path: 'D:\agent-2\_work\48\unzipped\*' to storage account: '***' in container: '$web' with blob prefix: ''
2023-03-23T20:57:11.2829679Z ##[command] & "AzCopy\AzCopy.exe" copy "D:\agent-2\_work\48\unzipped\*" "https://***.blob.core.windows.net/`$web"  --log-level=INFO --recursive
2023-03-23T20:57:11.3674171Z INFO: Scanning...
2023-03-23T20:57:11.3726322Z INFO: Authenticating to destination using Azure AD
2023-03-23T20:57:11.6319924Z INFO: AzCopy.exe: A newer version 10.17.0 is available to download
2023-03-23T20:57:11.6320551Z 
2023-03-23T20:57:11.9612580Z INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2023-03-23T20:57:11.9859681Z 
2023-03-23T20:57:11.9863288Z Job 11aab09c-9194-0943-4e2b-69d50ade80ec has started
2023-03-23T20:57:11.9864869Z Log file is located at: C:\Users\***\.azcopy\11aab09c-9194-0943-4e2b-69d50ade80ec.log
2023-03-23T20:57:11.9865670Z 
2023-03-23T20:57:11.9866037Z 
2023-03-23T20:57:14.0299238Z 0.0 %, 0 Done, 0 Failed, 85 Pending, 0 Skipped, 85 Total, 
2023-03-23T20:57:16.0201418Z 17.0 %, 13 Done, 0 Failed, 72 Pending, 0 Skipped, 85 Total, 2-sec Throughput (Mb/s): 16.8851
2023-03-23T20:57:16.0289937Z 
2023-03-23T20:57:16.0314754Z 
2023-03-23T20:57:16.0390409Z Job 11aab09c-9194-0943-4e2b-69d50ade80ec summary
2023-03-23T20:57:16.0392051Z Elapsed Time (Minutes): 0.0673
2023-03-23T20:57:16.0392356Z Number of File Transfers: 85
2023-03-23T20:57:16.0392873Z Number of Folder Property Transfers: 0
2023-03-23T20:57:16.0393145Z Total Number of Transfers: 85
2023-03-23T20:57:16.0393397Z Number of Transfers Completed: 85
2023-03-23T20:57:16.0393655Z Number of Transfers Failed: 0
2023-03-23T20:57:16.0394030Z Number of Transfers Skipped: 0
2023-03-23T20:57:16.0394397Z TotalBytesTransferred: 9665770
2023-03-23T20:57:16.0394716Z Final Job Status: Completed
2023-03-23T20:57:16.0394927Z 
2023-03-23T20:57:16.0415635Z Uploaded files successfully from source path: 'D:\agent-2\_work\48\unzipped\*' to storage account: '***' in container: '$web' with blob prefix: ''
2023-03-23T20:57:16.0428494Z ##[command] & "AzCopy\AzCopy.exe" logout
2023-03-23T20:57:16.1040099Z INFO: Logout succeeded.
2023-03-23T20:57:16.3837112Z INFO: AzCopy.exe: A newer version 10.17.0 is available to download
2023-03-23T20:57:16.3837402Z 
2023-03-23T20:57:16.4044036Z INFO: AzCopy.exe: A newer version 10.17.0 is available to download
2023-03-23T20:57:16.4099732Z 
2023-03-23T20:57:16.6047249Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2023-03-23T20:57:16.6479722Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2023-03-23T20:57:16.6851056Z 
2023-03-23T20:57:16.6852957Z 
2023-03-23T20:57:16.7648013Z ##[section]Finishing: Copy addin files to storage account

Relevant log output

I can see it's using the built-in mime mapping, but I want to use my own mapping.

2023-03-23T20:57:09.8587393Z  mime: D:\agent-2\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\5.218.0\MimeMapping.json

Aditional info

In my release I create a custom json file which contains my custom mime mapping, but it doesn't seem to be getting picked up by the task. I can see the version of azcopy.exe should be 10.12, which is supposed to read the file located at AZCOPY_CONTENT_TYPE_MAP, but it doesn't seem to be doing anything with it.

  - task: PowerShell@2
    displayName: 'Provide mime type map'
    inputs:
      targetType: 'inline'
      script: |
        $mimeTypeMapping = '{
          "MIMETypeMapping": {
            ".html": "text/html",
            ".css": "text/css",
            ".js": "text/javascript",
            ".xml": "application/xml",
            ".png": "image/png",
            ".ico": "image/vnd.microsoft.icon",
            ".map": "application/json",
            ".js.map": "application/json",
            ".json": "application/json"
          }
        }'

        $mimeTypeMapFilePath = '$(Pipeline.Workspace)/mimeTypeMapping.json'

        Out-File -FilePath $mimeTypeMapFilePath -InputObject $mimeTypeMapping

        Write-Host "##vso[task.setvariable variable=AZCOPY_CONTENT_TYPE_MAP]$mimeTypeMapFilePath"
celluj34 commented 1 year ago

In fact, it looks like it's just straight overwritten: https://github.com/microsoft/azure-pipelines-tasks/blob/42521e11655e3919e11976934375e021f4bc4896/Tasks/AzureFileCopyV5/Utility.ps1#L176

celluj34 commented 1 year ago

Hello, has anyone had time to look at this?

github-actions[bot] commented 10 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

celluj34 commented 10 months ago

Bump

TerraX3000 commented 6 months ago

Any solution for this issue? I set a pipeline variable for AZCOPY_CONTENT_TYPE_MAP to the file path to my mime type json file in my DevOps repo. This approach did not work either. My PowerShell script reads the environment variable with the path and then displays the file contents but AzureFileCopy does not use the mime type file for copying the files.

steps:
  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      script: |
        ls
        pwd
        ${env:AZCOPY_CONTENT_TYPE_MAP}
        cat ${env:AZCOPY_CONTENT_TYPE_MAP
}
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.236.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\9f84596b-c557-4e8f-867c-4a7cd70e1f30.ps1'"

    Directory: D:\a\1\s

Mode                 LastWriteTime         Length Name                                                                 
----                 -------------         ------ ----                                                                 
d-----         3/25/2024   1:34 PM                static                                                               
-a----         3/25/2024   1:34 PM            711 azure-pipeline.yaml                                                  
-a----         3/25/2024   1:34 PM             68 mime_types.json                                                      

Drive        : D
Provider     : Microsoft.PowerShell.Core\FileSystem
ProviderPath : D:\a\1\s
Path         : D:\a\1\s

mime_types.json
{
    "MIMETypeMapping": {
        ".webp": "image/webp"
    }
}

Finishing: PowerShell
celluj34 commented 6 months ago

I worked around this by using the az cli directly. Unfortunately I never got this working with this specific task.