search

aws / aws-tools-for-powershell

The AWS Tools for PowerShell lets developers and administrators manage their AWS services from the PowerShell scripting environment.
Apache License 2.0
235 stars 78 forks source link

Glacier Archive Description not working in Write-GLCArchive command #142

Closed shanecottey closed 1 year ago

shanecottey commented 4 years ago

When writing an archive into a Glacier vault with the AWSPowershell module command "Write-GLCArchive", no string that I pass into the -Description parameter is successfully saved with the written archive. I know the issue does not have to do with the API credentials in-use, because I can successfully upload and download the archives, as well as perform other administrative tasks in Glacier. I intended to pass in XML encoded meta-data for the Archive file being written (original file path & last write time), but even when trying something simple like "test" as the description and later reviewing the written Archive in a vault inventory file, the "ArchiveDescription" JSON field always comes back as an empty string "". I have temporarily worked around this by building a local index of the archive meta-data, but this doesn't make it easy to review and understand the vault inventory without having access to this index.

An example command for uploading a file with description-- Write-GLCArchive -VaultName "<vault name>" -FilePath "c:\test\hw.aws.txt" -Description "HelloWorldDescription"

This successfully returns FilePath, CheckSum, and ArchiveId results without any signs of errors or warnings--

FilePath = c:\test\hw.aws.txt ArchiveId = _HvCLiH6mx3f_llXfPS2xSwHSLkp9ZJ39yPT7v3MrdHhOTrBD0PTMjTuZU7TOliPuRCTTRkr2XuZOI25eTgcd3Uhaycfcqk9tUgraj5Yxe3-vU7ZX-IWF2QI9cfMLM9MYCbdk_VAZQ CheckSum = 442d14932dd27ddbb2b2d7e57130f7ea050147c775c...

When later reviewing that archive in an inventory file, this is what it shows up as-- {"ArchiveId":"_HvCLiH6mx3f_llXfPS2xSwHSLkp9ZJ39yPT7v3MrdHhOTrBD0PTMjTuZU7TOliPuRCTTRkr2XuZOI25eTgcd3Uhaycfcqk9tUgraj5Yxe3-vU7ZX-IWF2QI9cfMLM9MYCbdk_VAZQ", "ArchiveDescription":"", "CreationDate":"2020-02-10T19:16:17Z", "Size":58, "SHA256TreeHash":"442d14932dd27ddbb2b2d7e57130f7ea050147c775c..."}

Version info below: OS = Server 2012 R2 x64 PSVersion = 5.1.14409.1018 AWSPowerShell version = 4.0.2.0

This issue has also been posted in StackOverflow and AWS Developer forums, but so far has not gotten any response. https://stackoverflow.com/questions/60555899/glacier-archive-description-not-working-in-write-glcarchive-command https://forums.aws.amazon.com/thread.jspa?threadID=318204&tstart=0

netsurion-jsayre commented 4 years ago

Can anyone make a suggestion on this issue? I too am having it.

ashishdhingra commented 3 years ago

Appears to be reproducible. To get the archive description after executing Write-GLCArchive PowerShell command, following AWS CLI commands could be used:

aws glacier initiate-job --vault-name test --account-id - --job-parameters '{"Type": "inventory-retrieval"}'

aws glacier describe-job --vault-name test --account-id - --job-id <<JobId from initiate job command>>

aws glacier get-job-output --vault-name test --account-id - --job-id <<JobId from initiate job command>> output.json

Examining the output.json returns blank archive description as reported. Kindly note that the PowerShell code for Description parameter appears to apply for the single file. Not sure if the same description would be applied to archive description.

ashishdhingra commented 3 years ago

Not reproducible using AWS CLI.

Steps: (assumes we have already created S3 Glacier vault named test-vault)

netsurion-jsayre commented 3 years ago

Not reproducible using AWS CLI.

The issue is not the AWS CLI - the issue is with PowerShell, specifically the AWSPowerShell module. When uploading an archive via PowerShell (again, not AWS CLI), the description is not retained.

ashishdhingra commented 3 years ago

Not reproducible using AWS CLI.

The issue is not the AWS CLI - the issue is with PowerShell, specifically the AWSPowerShell module. When uploading an archive via PowerShell (again, not AWS CLI), the description is not retained.

Yes, it is a PowerShell issue. AWS CLI (no) reproduction steps are just for developer's reference.

github-actions[bot] commented 2 years ago

We have noticed this issue has not recieved attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

shanecottey commented 2 years ago

Was this issue ever resolved in a newer version of the AWSPowerShell module?

giammin commented 1 year ago

no, it still does not work

ashishdhingra commented 1 year ago

Somehow when uploading the archive using below code:

using Amazon;
using Amazon.Glacier;
using Amazon.Glacier.Model;
using Amazon.Glacier.Transfer;
using Amazon.Runtime;

ArchiveTransferManager manager;
string archiveId;

//Set the vault name you want to use here.
string vaultName = "testvault";

// Set the file path for the file you want to upload here.
string filePath = @"<<absolute-path-to-file>>";

AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;
AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

using (manager = new ArchiveTransferManager(RegionEndpoint.USEast2))
{
    try
    {
        var uploadResult = await manager.UploadAsync(vaultName, "Test Archive Description with response logging", filePath);
        archiveId = uploadResult.ArchiveId;
        Console.WriteLine("Upload successful. Archive Id : {0}  Checksum : {1}", uploadResult.ArchiveId, uploadResult.Checksum);
    }
    catch (AmazonGlacierException e)
    {
        Console.WriteLine(e.Message);
    }
    catch (AmazonServiceException e)
    {
        Console.WriteLine(e.Message);
    }
}

generates the below response logging:

Amazon Information: 0 : UserCrypto is not supported.  This may be due to use of a non-Windows operating system or Windows Nano Server, or the current user account may not have its profile loaded. Unable to load shared library 'Crypt32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libCrypt32.dll, 0x0001): tried: 'libCrypt32.dll' (no such file), '/usr/local/lib/libCrypt32.dll' (no such file), '/usr/lib/libCrypt32.dll' (no such file), '/Users/ashdhin/dev/repros/GlacierArchiveDescriptionTest/GlacierArchiveDescriptionTest/bin/Debug/net6.0/libCrypt32.dll' (no such file)
Amazon Information: 0 : The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
Amazon Information: 1 : The environment variable AWS_MAX_ATTEMPTS was not set with a value.
Amazon Information: 2 : The environment variable AWS_RETRY_MODE was not set with a value.
Amazon Information: 3 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
Amazon Information: 4 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
Amazon Information: 5 : The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
Amazon Information: 6 : The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
Amazon Information: 0 : There is no defaults_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 1 : There is no endpoint_discovery_enabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 2 : There is no retry_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 3 : There is no max_attempts set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 4 : There is no ec2_metadata_service_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 5 : There is no ec2_metadata_service_endpoint_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 6 : There is no use_dualstack_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 7 : There is no use_fips_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 0 : Resolved DefaultConfigurationMode for RegionEndpoint [us-east-2] to [Legacy].
Amazon Information: 0 : Starting a process with the following ProcessInfo: UseShellExecute - False RedirectStandardError - True, RedirectStandardOutput - True, CreateNoWindow - True
Amazon Information: 1 : Process started
Amazon Information: 0 : Process ends with exitcode - 0
Amazon Verbose: 2 : Double encoded /{accountId}/vaults/{vaultName}/archives with endpoint https://glacier.us-east-2.amazonaws.com/ for canonicalization: /-/vaults/testvault/archives
Amazon Verbose: 0 : Received response (truncated to 1024 bytes): []
Amazon Information: 1 : Request metrics: AsyncCall = True; CanonicalRequest = POST\n/-/vaults/testvault/archives\n\ncontent-length:3705\ncontent-type:binary/octet-stream\nhost:glacier.us-east-2.amazonaws.com\nuser-agent:aws-sdk-dotnet-coreclr/3.7.100.3 aws-sdk-dotnet-core/3.7.100.3 .NET_Core/6.0.8 OS/Darwin_21.6.0_Darwin_Kernel_Version_21.6.0:_Mon_Aug_22_20:17:10_PDT_2022;_root:xnu-8020.140.49~2/RELEASE_X86_64 ClientAsync ArchiveTransferManager/SingleUpload\nx-amz-api-version:2012-06-01\nx-amz-archive-description:Test Archive Description with response logging\nx-amz-content-sha256:f0704cb2882c19ced212110fbd7c6f14199e3de85f8504dd3844607a8f3a0d6c\nx-amz-date:20221028T001554Z\nx-amz-glacier-version:2012-06-01\nx-amz-security-token:<<security-token>>\n\ncontent-length;content-type;host;user-agent;x-amz-api-version;x-amz-archive-description;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-security-token;x-amz-sha256-tree-hash\nf0704cb2882c19ced212110fbd7c6f14199e3de85f8504dd3844607a8f3a0d6c; StringToSign = AWS4-HMAC-SHA256\n20221028T001554Z\n20221028/us-east-2/glacier/aws4_request\n7b69a813c7a0a1dcc8486b3ffd451612f1a55389a0a4573424b44e44c99050fc; ServiceName = Amazon.Glacier; ServiceEndpoint = https://glacier.us-east-2.amazonaws.com/; MethodName = UploadArchiveRequest; StatusCode = Created; BytesProcessed = 2; AWSRequestID = SUkvYQM9AbjsIbtiluLDBbKIohwkBOBrpCLx_S3_EHHW1rQ; CredentialsRequestTime = 00:00:13.5026836; RequestSigningTime = 00:00:00.0914661; HttpRequestTime = 00:00:00.8554563; ResponseUnmarshallTime = 00:00:00.0009691; ResponseProcessingTime = 00:00:00.0186403; ClientExecuteTime = 00:00:14.6126253; 
Upload successful. Archive Id : _mwh8MNuWXN6kUOi1FtI2zQKLnpDwNYC-2oBTSkLEeMXR98PVrO8NkamjcpfvrrOsvNc8u-CS2XF_YcLt_tRChJ3T-YDR6eRCYXO8zMZ2PASMJTsXU74AGswaM2EJZK-DnTmMM0K7w  Checksum : f0704cb2882c19ced212110fbd7c6f14199e3de85f8504dd3844607a8f3a0d6c

Notice the signed header x-amz-archive-description in the above log. Hence, the archive description is set correctly in this case (refer https://github.com/aws/aws-sdk-net/issues/2462 for details).

Enabling response logging in PowerShell does not log x-amz-archive-description as one of the signed headers. The below code:

Add-AWSLoggingListener MyAWSLogs <<absolute-path-to-log-file>>
Set-AWSResponseLogging Always
Enable-AWSMetricsLogging

Write-GLCArchive -VaultName "testvault" -FilePath <<absolute-path-of-file-to-upload>> -Description "Hello World Description" -Region "us-east-2"

generates the following response logging:

Amazon Information: 0 : Resolved DefaultConfigurationMode for RegionEndpoint [us-east-2] to [Legacy].
Amazon Information: 0 : Starting a process with the following ProcessInfo: UseShellExecute - False RedirectStandardError - True, RedirectStandardOutput - True, CreateNoWindow - True
Amazon Information: 1 : Process started
Amazon Information: 0 : Process ends with exitcode - 0
Amazon Verbose: 2 : Double encoded /{accountId}/vaults/{vaultName}/archives with endpoint https://glacier.us-east-2.amazonaws.com/ for canonicalization: /-/vaults/testvault/archives
Amazon Verbose: 0 : Received response (truncated to 1024 bytes): []
Amazon Information: 1 : Request metrics: AsyncCall = True; CanonicalRequest = POST\n/-/vaults/testvault/archives\n\ncontent-length:3705\ncontent-type:binary/octet-stream\nhost:glacier.us-east-2.amazonaws.com\nuser-agent:AWSPowerShell.Common/4.1.194.0 .NET_Core/6.0.8 OS/Darwin_21.6.0_Darwin_Kernel_Version_21.6.0:_Mon_Aug_22_20:17:10_PDT_2022;_root:xnu-8020.140.49~2/RELEASE_X86_64 PowerShellCore/7.-1 ClientAsync ArchiveTransferManager/SingleUpload\nx-amz-api-version:2012-06-01\nx-amz-content-sha256:f0704cb2882c19ced212110fbd7c6f14199e3de85f8504dd3844607a8f3a0d6c\nx-amz-date:20221028T001118Z\nx-amz-glacier-version:2012-06-01\nx-amz-security-token:<<security-token>>\n\ncontent-length;content-type;host;user-agent;x-amz-api-version;x-amz-content-sha256;x-amz-date;x-amz-glacier-version;x-amz-security-token;x-amz-sha256-tree-hash\nf0704cb2882c19ced212110fbd7c6f14199e3de85f8504dd3844607a8f3a0d6c; StringToSign = AWS4-HMAC-SHA256\n20221028T001118Z\n20221028/us-east-2/glacier/aws4_request\n81f4381ec358ce237d20b9b56aa31894fae2851444901e335fb7c53d34cbdea3; ServiceName = Amazon.Glacier; ServiceEndpoint = https://glacier.us-east-2.amazonaws.com/; MethodName = UploadArchiveRequest; StatusCode = Created; BytesProcessed = 2; AWSRequestID = Cywcp0pNkvymQusI3qE1rj0oAnEJd9vMK_yt3DD5dI2-YYA; CredentialsRequestTime = 00:00:11.8984179; RequestSigningTime = 00:00:00.0019313; HttpRequestTime = 00:00:00.6331241; ResponseUnmarshallTime = 00:00:00.0000597; ResponseProcessingTime = 00:00:00.0011880; ClientExecuteTime = 00:00:12.6060552;

TODO: Figure out where archive description is getting lost when executing Write-GLCArchive cmdlet.

giammin commented 1 year ago

maybe it is not read from the command line? or something overwrite it

giammin commented 1 year ago

@ashishdhingra maybe the problem is in this lines:

#if DESKTOP
                    var result = transferManager.Upload(vaultName, description, filePath, uploadOptions);
#else
                    var result = transferManager.UploadAsync(vaultName, description, filePath, uploadOptions).GetAwaiter().GetResult();
#endif

in your test you used the async version UploadAsync, could you test transferManager.Upload

ashishdhingra commented 1 year ago

The issue is fixed in AWS.Tools.Glacier 4.1.199.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

giammin commented 1 year ago

The issue is fixed in AWS.Tools.Glacier 4.1.199.

it works! thanks