generate a presigned URL to file in S3

[giuspen]

Is is possible somehow to generate a presigned URL to file in S3 with the aws-iot-device-sdk-embedded-C?

I've been looking at the demo http_demo_s3_download which I can run successfully.

I tried to add in http_demo_s3_download.c, just after

    if( returnStatus == true )
    {
        LogInfo( ( "The file is %d bytes long.", ( int32_t ) *pFileSize ) );

The following code:

        char ota_temp_url[2048] = "https://" \
                                  AWS_S3_ENDPOINT \
                                  AWS_S3_URI_PATH \
                                  "?response-content-disposition=inline";
        strcat(ota_temp_url, "&X-Amz-Security-Token=");
        strncat(ota_temp_url, pSecurityToken, securityTokenLen);
        strcat(ota_temp_url, "&X-Amz-Algorithm=");
        strcat(ota_temp_url, SIGV4_AWS4_HMAC_SHA256);
        strcat(ota_temp_url, "&X-Amz-Date=");
        strncat(ota_temp_url, pDateISO8601, SIGV4_ISO_STRING_LEN);
        strcat(ota_temp_url, "&X-Amz-SignedHeaders=host");
        strcat(ota_temp_url, "&X-Amz-Expires=3600");
        strcat(ota_temp_url, "&X-Amz-Credential=");
        {
            char* pchar_start = strstr(pSigv4Auth, " Credential=");
            if (pchar_start) {
                char* pchar_end = strstr(pchar_start, ",");
                if (pchar_end) {
                    strncat(ota_temp_url, pchar_start+12, pchar_end-pchar_start-12);
                }
            }
        }
        strcat(ota_temp_url, "&X-Amz-Signature=");
        strncat(ota_temp_url, signature, signatureLen);
        LogInfo( ( "ota_temp_url=%s", ota_temp_url ) );

This is generating a URL with the correct syntax but I'm failing to understand what's wrong as the answer I get is:

<Error>
<Code>InvalidToken</Code>
<Message>The provided token is malformed or otherwise invalid.</Message>
<Token-0>IQoJb3JpZ2luX2VjEMb//////////wEaDGV1LWNlbnRyYWwtMSJGMEQCIBrv7gABmZniPAwon17t1bJQP7f7QilcTJU3QM0 XKVAiBEs/69RoSRWLOlM3uwtIhQA7kz8rnJmOzREgCzyumHiyrnAwi///////////8BEAAaDDM0NjQ0NDczNTIyNiIM08c2idyZjLoTyrxEKrsDSqeQAZ// oou4ENh2LylLXIVS9tlsB82Jg96 v2a jtM81N fXia5DD9v/IthIJrm8E TDkmkg09L7nwpgyB5ombnXfiv6jNaZTjR W3GR/NGh33nLCh9eywgfovbf2BD8TKgBU6ke74TGuxm3phFzW2K1OgCtP4oDUq94e432SKaZJsiD97jA20MX8IfVwD6xwBX0N25RHilQDipgl/DqeR3hvZqlZmIzIvzNvJ6OStuhMIGX5uKg4Dy4/zwNWfIGGr6AhR6XT9e3qESihLXWapMb0kVVxupVOteqdgbGpl3NbmJdBWVQ1YhKfVhXCXo8fWdHtFomCxJfu15o4YKMWwLPWPADu/TTI2DXdcCYY32cyVy54DjmEQnL5kXI7QOv3c3mS1FE56XlUK8eoR4rwXL9cUXsNrbCTPFOeasxlohZ4e jVASLl RQx3Xh2v1dvPKCGtywBgc0MLXvPiAWxnpnraGrpsksXXY6DsWYbIsnHBuomblCzF5N1T85EBP2VQbsUo1U9ez xL8WVgKZqUaF4kxTwZEBF5FaZelFY9Lle ActCn2C41yRJ0XSSWrdasSI t77YOcww7bbqAY6mwGn6MspGaXNR AWi5qQbcH6Hxee4s L74ZEcGrnK91IcuJ djHfLhLRUJIEUJLruGnkgqdP9DZmcv1TmLrr2vByWkFG867SSF2slWzbtwWIrML/7jxI KTSou5wPHQC3fvjFboQ5f0ub5wUott/bmneKo7Np2QtjmpiKQi7JHazcP2vRxBmzQ6JpLBXWsXuDVzeYLfsIgvPV/mG0A==</Token-0>
<RequestId>EE873P33F22DW1K6</RequestId>
<HostId>cljT52u0sAuOuzSohPGVOW9hxaCMuHd/hOEEkCMj8hTWf9guhbDzwC4J5W49QTQeww3kgrB2e3Yeax5ieI2MCw==</HostId>
</Error>

Can anybody help me?

[chinglee-iot]

Hi giuspen, Thank you for your question. We will look into your question and discuss with you here.

[giuspen]

I was eventually able to modify the demo to presign the URL. I've been working on https://github.com/giuspen/aws-iot-device-sdk-embedded-C/tree/GP_http_demo_s3_download_test_signature If of interest I can prepare a separate demo in the next days and submit a PR but that is going to require a change/PR also on a submodule https://github.com/giuspen/SigV4-for-AWS-IoT-embedded-sdk/tree/GP_SupportForPresignedURLs

[giuspen]

While I have the signature working (checking it against aws s3 presign), still it complains with

<Error>
<Code>InvalidAccessKeyId</Code>
<Message>The AWS Access Key Id you provided does not exist in our records.</Message>
<AWSAccessKeyId>ASIAVBKNXEL5BG7MXQYD</AWSAccessKeyId>
<RequestId>BDXH8ATGQKT0C9R3</RequestId>
<HostId>7DmSSvMCANShEmY37IpVT9JB4q39Z4AbNiLCB0U3DheaEm4IlBsjQJJly7pnHYaUU6corYHtcBU=</HostId>
</Error>

Is the presign offline or requires a particular message to the server?

The AccessKeyId is the one I get in the S3 download demo, which has a temporary validity, why the server would say that it doesn't exist :(

[giuspen]

In a new branch https://github.com/giuspen/aws-iot-device-sdk-embedded-C/tree/GP_http_demo_s3_download_test_signature2 the file is downloaded correctly as per the demo, then the presigned URL is generated with a correct signature and the same AccessKeyId that it is invalid though when using the URL

[chinglee-iot]

@giuspen

Thank you for your information and contribution. Generating a presigned URL to S3 file with this repo is a new demo to me so probably it will take some time to solve the problem with you. At the same time, I will also consult with experienced member.

Feel free to create a PR to https://github.com/giuspen/SigV4-for-AWS-IoT-embedded-sdk/tree/GP_SupportForPresignedURLs. We would like to discuss with you there.

It takes some consideration to add new demo in this repository. I will bring back this demo for discussion.

[giuspen]

thanks @chinglee-iot I'm also adding the log that I also put on the aws re:Post forum ( https://repost.aws/questions/QULTkclXUuQGWxVNiQAvIDOw/generation-of-presigned-url-to-s3-with-temporary-credentials-invalidaccesskeyid )

[INFO] [DEMO] [http_demo_s3_download.c:1752] HTTP Client Synchronous S3 download demo using temporary credentials fetched from iot credential provider: c10nfdpw0ux517.credentials.iot.eu-central-1.amazonaws.com
[INFO] [DEMO] [http_demo_s3_download.c:910] Establishing a TLS session with c10nfdpw0ux517.credentials.iot.eu-central-1.amazonaws.com:443.
[INFO] [DEMO] [http_demo_s3_download.c:766] AWS IoT credentials will expire after this timestamp: 2023-10-02T13:30:09Z. 
[INFO] [DEMO] [http_demo_s3_download.c:650] AWS IoT credential provider response: HTTP/1.1 200 OK content-type: application/json content-length: 1165 date: Mon, 02 Oct 2023 12:30:09 GMT x-amzn-RequestId: fd0bb5c9-754f-9486-6cfa-2529899e17c8
{"credentials":{"accessKeyId":"ASIAVBKNXEL5HP2B4VXX","secretAccessKey":"<redacted>","sessionToken":"IQoJb3JpZ2luX2VjEA0aDGV1LWNlbnRyYWwtMSJHMEUCIFsKZ0d7JLu+P1S0AfpaXUGuW5/agg0/3+joQt+D2oirAiEA+fCfmAqo9zt8Xb5CcuJj7He77UQQcgq/HMaZNJljQKcq3gMIFhAAGgwzNDY0NDQ3MzUyMjYiDCk8oHzTPPEymfx9Kiq7AwvmDkvcH5RPb9RWtAoci/WO/UhI4D1CQ5kg2X7KEUbpdk/ig8W6hSMhEcFWYEfMNpryhicDB/XoJzpjfvPxccOewA6Rd/y+2qFzNOOc2Cba3GtovdGctYJBtvex6du367kJoYks6PtaKpxFXyA6oEHWQ2X0yy9Ai+bZdPX9woHEJOQ7x9Orr6msJo47wSFgF9R5JlPk2n1mvLuI1kQlzPbgakTSwkSYb0hR91Sbd9V9kWQzQsK1GI3TktJ9YkPM0EIP5FMlLtwBPa0IYNMpHT1mz5LnXg/Imoepb6sZtsfW/M1LhkbgL//8sTeE5HHUXIL0RbRRSEQSctFPETIozfmKDp+yGNeOgFXDKwuY7aeKTzw1URnKHSM9mN5d5BVCdIjFu4obUgOLrL1tSVK3ai+54d1pXplNkcT1RDuZUKPvdj1eNDb46Ykd70LbtuQCHZNJQLeIxOQ/XbH3slf49UXp6Gwr8ENk888U+w8W3Emi1uZ6OkM4X00DKZaUksQCH1/my7tAwOP/RZNypXiQj6d33COeu2W92NvTt74xH8XrUEBOxxAQuDTasOTf8SK4E+y7Q9GoB0tb9CQiMNHv6qgGOpoB3LgWRNLccumU7fynfuXkdXoi3utGH7UpWqs8zz2mImgSJccDFg8NEyT9x+yhdcitTiJ7lugAFNgKg1qTpphL6i0CRDExX6JyYYwXU1l3BxSxLMSehYlLWpJV+/SCOVqur9ps+7dC3M3+hjuSG3jusANCGbbINzink/mQmXSHDnGL8zcbV6YpLPEH+DVXuz4cFW3FT2jbkjuV4g==","expiration":"2023-10-02T13:30:09Z"}}.
[INFO] [DEMO] [http_demo_s3_download.c:961] Establishing a TLS session with otahalo.s3.eu-central-1.amazonaws.com:443.
[INFO] [DEMO] [http_demo_s3_download.c:1478] Getting file object size from host...
[INFO] [DEMO] [http_demo_s3_download.c:1652] Received successful response from server (Status Code: 206).
[INFO] [DEMO] [http_demo_s3_download.c:1698] The file is 58 bytes long.
[INFO] [DEMO] [http_demo_s3_download.c:1176] Downloading bytes 0-57, out of 58 total bytes, from otahalo.s3.eu-central-1.amazonaws.com...:
[INFO] [DEMO] [http_demo_s3_download.c:1206] Response Body: { "filename"="update-bundle-imx8mnevk-v01-00-01.raucb" }
[INFO] [DEMO] [http_demo_s3_download.c:1307] Getting presigned URL...
[INFO] [DEMO] [http_demo_s3_download.c:1410] ota_temp_url= https://otahalo.s3.eu-central-1.amazonaws.com/ota-v01-00-01.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAVBKNXEL5HP2B4VXX%2F20231002%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20231002T123009Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEA0aDGV1LWNlbnRyYWwtMSJHMEUCIFsKZ0d7JLu%2BP1S0AfpaXUGuW5%2Fagg0%2F3%2BjoQt%2BD2oirAiEA%2BfCfmAqo9zt8Xb5CcuJj7He77UQQcgq%2FHMaZNJljQKcq3gMIFhAAGgwzNDY0NDQ3MzUyMjYiDCk8oHzTPPEymfx9Kiq7AwvmDkvcH5RPb9RWtAoci%2FWO%2FUhI4D1CQ5kg2X7KEUbpdk%2Fig8W6hSMhEcFWYEfMNpryhicDB%2FXoJzpjfvPxccOewA6Rd%2Fy%2B2qFzNOOc2Cba3GtovdGctYJBtvex6du367kJoYks6PtaKpxFXyA6oEHWQ2X0yy9Ai%2BbZdPX9woHEJOQ7x9Orr6msJo47wSFgF9R5JlPk2n1mvLuI1kQlzPbgakTSwkSYb0hR91Sbd9V9kWQzQsK1GI3TktJ9YkPM0EIP5FMlLtwBPa0IYNMpHT1mz5LnXg%2FImoepb6sZtsfW%2FM1LhkbgL%2F%2F8sTeE5HHUXIL0RbRRSEQSctFPETIozfmKDp%2ByGNeOgFXDKwuY7aeKTzw1URnKHSM9mN5d5BVCdIjFu4obUgOLrL1tSVK3ai%2B54d1pXplNkcT1RDuZUKPvdj1eNDb46Ykd70LbtuQCHZNJQLeIxOQ%2FXbH3slf49UXp6Gwr8ENk888U%2Bw8W3Emi1uZ6OkM4X00DKZaUksQCH1%2Fmy7tAwOP%2FRZNypXiQj6d33COeu2W92NvTt74xH8XrUEBOxxAQuDTasOTf8SK4E%2By7Q9GoB0tb9CQiMNHv6qgGOpoB3LgWRNLccumU7fynfuXkdXoi3utGH7UpWqs8zz2mImgSJccDFg8NEyT9x%2ByhdcitTiJ7lugAFNgKg1qTpphL6i0CRDExX6JyYYwXU1l3BxSxLMSehYlLWpJV%2B%2FSCOVqur9ps%2B7dC3M3%2BhjuSG3jusANCGbbINzink%2FmQmXSHDnGL8zcbV6YpLPEH%2BDVXuz4cFW3FT2jbkjuV4g%3D%3D%22%2C%22expiration%22%3A%222023-10-02T13%3A30%3A09Z%22%7D%7D&X-Amz-Signature=fd138ce95b8bf62db8ba1a0cf1aea3147a48e4c95231b44e2c4a6a30aebd52b7
[INFO] [DEMO] [http_demo_s3_download.c:1856] Demo iteration 1 is successful.
[INFO] [DEMO] [http_demo_s3_download.c:1875] Demo completed successfully.

NOW If I use the presigned URL:

<Error>
  <Code>InvalidAccessKeyId</Code>
  <Message> The AWS Access Key Id you provided does not exist in our records. </Message>
  <AWSAccessKeyId>ASIAVBKNXEL5HP2B4VXX</AWSAccessKeyId>
  <RequestId>1ZXWN5V7S7N41VHX</RequestId>
  <HostId> 3lXR0Y18JE21ljldrxdoXZrZ1PCR6kC5MAEp9G+RcVms+fA1kRpn0S8aVJ9mFP/6ydlvCXHlFs0= </HostId>
</Error>

[ydhuang28]

@giuspen We're seeing lots of access key ids, tokens, etc. in your log output; a friendly reminder to remove these when posting logs (We've already redacted the secretAccessKey).

[giuspen]

Thanks @ydhuang28 those are anyway temporary credentials valid 1h in a test only environment

[chinglee-iot]

@giuspen

The X-Amz-Security-Token in your ota_temp_url look different. There is an extra string "%22%2C%22expiration%22%3A%222023-10-02T13%3A30%3A09Z%22%7D%7D" at the end. You may compared your presigned URL result with the presigned URL generated by aws cli with temporary credential.

[INFO] [DEMO] [http_demo_s3_download.c:1410] ota_temp_url= https://otahalo.s3.eu-central-1.amazonaws.com/ota-v01-00-01.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAVBKNXEL5HP2B4VXX%2F20231002%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20231002T123009Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEA0aDGV1LWNlbnRyYWwtMSJHMEUCIFsKZ0d7JLu%2BP1S0AfpaXUGuW5%2Fagg0%2F3%2BjoQt%2BD2oirAiEA%2BfCfmAqo9zt8Xb5CcuJj7He77UQQcgq%2FHMaZNJljQKcq3gMIFhAAGgwzNDY0NDQ3MzUyMjYiDCk8oHzTPPEymfx9Kiq7AwvmDkvcH5RPb9RWtAoci%2FWO%2FUhI4D1CQ5kg2X7KEUbpdk%2Fig8W6hSMhEcFWYEfMNpryhicDB%2FXoJzpjfvPxccOewA6Rd%2Fy%2B2qFzNOOc2Cba3GtovdGctYJBtvex6du367kJoYks6PtaKpxFXyA6oEHWQ2X0yy9Ai%2BbZdPX9woHEJOQ7x9Orr6msJo47wSFgF9R5JlPk2n1mvLuI1kQlzPbgakTSwkSYb0hR91Sbd9V9kWQzQsK1GI3TktJ9YkPM0EIP5FMlLtwBPa0IYNMpHT1mz5LnXg%2FImoepb6sZtsfW%2FM1LhkbgL%2F%2F8sTeE5HHUXIL0RbRRSEQSctFPETIozfmKDp%2ByGNeOgFXDKwuY7aeKTzw1URnKHSM9mN5d5BVCdIjFu4obUgOLrL1tSVK3ai%2B54d1pXplNkcT1RDuZUKPvdj1eNDb46Ykd70LbtuQCHZNJQLeIxOQ%2FXbH3slf49UXp6Gwr8ENk888U%2Bw8W3Emi1uZ6OkM4X00DKZaUksQCH1%2Fmy7tAwOP%2FRZNypXiQj6d33COeu2W92NvTt74xH8XrUEBOxxAQuDTasOTf8SK4E%2By7Q9GoB0tb9CQiMNHv6qgGOpoB3LgWRNLccumU7fynfuXkdXoi3utGH7UpWqs8zz2mImgSJccDFg8NEyT9x%2ByhdcitTiJ7lugAFNgKg1qTpphL6i0CRDExX6JyYYwXU1l3BxSxLMSehYlLWpJV%2B%2FSCOVqur9ps%2B7dC3M3%2BhjuSG3jusANCGbbINzink%2FmQmXSHDnGL8zcbV6YpLPEH%2BDVXuz4cFW3FT2jbkjuV4g%3D%3D%22%2C%22expiration%22%3A%222023-10-02T13%3A30%3A09Z%22%7D%7D&X-Amz-Signature=fd138ce95b8bf62db8ba1a0cf1aea3147a48e4c95231b44e2c4a6a30aebd52b7

The other thing observed from your code is that "strcat" is used instead of HTTPClient_AddHeader and getHeaderStartLocFromHttpRequest is called before the all the headers are added. This may cause the signature calculation error.

[giuspen]

@chinglee-iot you're a star, thank you so much!

It is working now. I will prepare a proper new demo in the next days and submit a PR

[giuspen]

note that there is an additional submodule to patch to be able to run the demo on the branch GP_http_demo_s3_download_test_signature2 of https://github.com/giuspen/aws-iot-device-sdk-embedded-C.git , for now I have done only a temporary macro out:

penoneg@cinnamon:~/git/aws-iot-device-sdk-embedded-C/libraries/standard/coreHTTP$ g diff
diff --git a/source/core_http_client.c b/source/core_http_client.c
index 63836b5..c7f008d 100644
--- a/source/core_http_client.c
+++ b/source/core_http_client.c
@@ -1622,7 +1622,7 @@ HTTPStatus_t HTTPClient_InitializeRequestHeaders( HTTPRequestHeaders_t * pReques
                                          pRequestInfo->pPath,
                                          pRequestInfo->pathLen );
     }
-
+#if 0
     if( returnStatus == HTTPSuccess )
     {
         /* Write "User-Agent: <Value>". */
@@ -1632,7 +1632,7 @@ HTTPStatus_t HTTPClient_InitializeRequestHeaders( HTTPRequestHeaders_t * pReques
                                   HTTP_USER_AGENT_VALUE,
                                   HTTP_USER_AGENT_VALUE_LEN );
     }
-
+#endif
     if( returnStatus == HTTPSuccess )
     {
         /* Write "Host: <Value>". */

[chinglee-iot]

Free free to create a patch pull request If you are mentioning the SigV4 submodule. We would like to discuss with you there.

I also observed that there may be some errors in the signature calculation. Posting here again just to make sure you noticed

The other thing observed from your code is that "strcat" is used instead of HTTPClient_AddHeader and getHeaderStartLocFromHttpRequest is called before the all the headers are added. This may cause the signature calculation error.

[giuspen]

@chinglee-iot you also have to modify coreHTTP manually with the above diff -> no HTTP_USER_AGENT_VALUE Plus I found another issue with the date I'm a bit overwhelmed from work but will try and push the fix shortly

[giuspen]

@chinglee-iot I have now fixed both my branches if you want to test. GP_http_demo_s3_download_test_signature -> only generates the presigned URL, no download GP_http_demo_s3_download_test_signature2 -> generates the presigned URL, and also downloads I will proceed with the pull requests shortly anyway

[giuspen]

I have now created the required PR to the two submodules coreHTTP and SigV4.

If those will be merged, then I will proceed to create a proper new demo with a new name such as http_demo_s3_presigned_url and submit a PR to this repo.