Write-S3Object ignores HeaderCollection argument when uploading folders

iforapsy commented 4 years ago

When I try to upload a folder to a key prefix using Write-S3Object, the -HeaderCollection argument seems to be silently ignored. The folder is successfully uploaded, but none of the S3 objects have the metadata headers that I specified.

Expected Behavior

The files are uploaded to S3 and they have the metadata headers that I specified in the HeaderCollection argument.

Current Behavior

The files are uploaded but without the headers that I specified.

Possible Solution

Unsure

Steps to Reproduce (for bugs)

Write-S3Object -Folder my-folder -BucketName my-bucket -KeyPrefix my-prefix/ -HeaderCollection @{'Cache-Control'='max-age=12345'}

Context

I am trying to use the Write-S3Object cmdlet to recursively upload a directory tree to my S3 bucket. This S3 bucket will be used as the origin for a CloudFront distribution so that is why I want to set special headers on the S3 objects, most importantly the Cache-Control header.

Your Environment

I am running Windows 10 Enterprise version 1903.

PS C:\> Get-AWSPowerShellVersion

AWS Tools for Windows PowerShell
Version 4.0.3.0
Copyright 2012-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET
Core Runtime Version 3.3.104.38
Copyright 2009-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Release notes: https://github.com/aws/aws-tools-for-powershell/blob/master/CHANGELOG.md

This software includes third party software subject to the following copyrights:
- Logging from log4net, Apache License
[http://logging.apache.org/log4net/license.html]

PS C:\> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.18362.752
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.18362.752
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

matteo-prosperi commented 4 years ago

This is a reasonable request. Marking as feature request.

iforapsy commented 4 years ago

I would classify it as more of a bug since nowhere in the documentation does it state that HeaderCollection is silently discarded when uploading a folder. In fact, the docs explicitly list -HeaderCollection as a valid argument to the UploadFolder variant of the cmdlet.

kowenhouston commented 4 years ago

I suspect I may be having the same issue but with the Copy-S3Object cmdlet. I'm happy to open another issue but I thought maybe this might be the same thing?

Copy-S3Object -BucketName my-bucket -DestinationBucket my-bucket -Key 'index.html' -DestinationKey 'index.html' -HeaderCollection @{"Cache-Control" = "no-cache"}

Running the command actually results in an error:

Copy-S3Object : This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata,
storage class, website redirect location or encryption attributes.

When modifying the destination key to a unique key results in the object being copied as expected, however the header collection, again, is being silently ignored.

pauladler commented 3 years ago

I also have exactly the same use case with S3 and CloudFront and am experiencing the same bug.

PS C:> Get-AWSPowerShellVersion

ashishdhingra commented 3 years ago

Write-S3Object for -Folder parameter uses TransferUtilityUploadDirectoryRequest, which (and some related classes) does not currently have any property for specifying Headers. This is most likely a change in AWSSDK.S3 followed by Write-S3Object CmdLet.

EDIT: Write-S3Object CmdLet uses UploadFolderToS3 to upload folder using the TransferUtilityUploadDirectoryRequest. Before executing the TransferUtility.UploadDirectory(), it executes AmazonS3Helper.SetExtraRequestFields(request, cmdletContext) to set the TransferUtilityUploadDirectoryRequest. UploadDirectoryFileRequestEvent, which appears to execute SetMetadataAndHeaders(uploadRequest, cmdletContext.Metadata, cmdletContext.Metadata) for each file. Notice it uses cmdletContext.Metadata for headers as well, which could be the reason on why headers are ignored.

ashishdhingra commented 3 years ago

This is fixed in AWS.Tools.S3 4.1.13.0.

github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-tools-for-powershell

Write-S3Object ignores HeaderCollection argument when uploading folders #144

⚠️COMMENT VISIBILITY WARNING⚠️