Signature fails when path is URL encoded with CF-2016

[doodi-v1]

I was fighting with a AWS signature does not match issue and narrowed it down to a specific line in coldfusion.cfc:20. Original line is:

var fullPath = utils.encodeUrl( path, false ) & ( !queryParams.isEmpty() ? ( '?' & utils.parseQueryParams( queryParams ) ) : '' );

Changing to:

var fullPath = path & ( !queryParams.isEmpty() ? ( '?' & utils.parseQueryParams( queryParams ) ) : '' );

fixes the issue.

I was posting a request to the execute-api service at AWS. My URI is: https://someletters.execute-api.us-west-2.amazonaws.com/beta/@connections/a0abIc59PHcCJQQ=

[doodi-v1]

Two issues in the URI caused the signature to fail: the @ symbol and the trailing = symbol. AWS receives these correctly with the change. Without the change, AWS double encodes the character.

[jcberquist]

This is interesting. @ and = are both reserved characters and so I would expect them to be encoded in the URL. With the change you note above, ACF does not encode those characters. When I test paths with those characters, encoding them seems to work. I have not tested the execute-api service, however. Could you say more about what you mean by "AWS double encoded the character"?

[doodi-v1]

Here is the original error response from AWS that I was receiving (luckily, they do a good job of documenting the failure in the message):

The Canonical String for this request should have been 'POST /beta/%2540connections/ax04CfGcPHcCJkg%253D

host:sometext.execute-api.us-west-2.amazonaws.com\nx-amz-date:20190605T175310Z\n\nhost;x-amz-date\n9779345914b767f358db1224bd6bfa2fd15c1ba52d2ecadf3fca3670935e021b'

Note the %2540 and %253D - double-URL encoded @ and =

[doodi-v1]

I've been told by my IT that one of the last CF updates greatly increased CF URL encoding - tags that didn't URL encode were now encoding automatically. Maybe it's a CF version issue.

[doodi-v1]

By the way, great job with the library! I would still be researching how to properly sign AWS requests if I did not have your code as a starting point. Kudos!!!

[jcberquist]

I do wonder if it is an issue with a specific version of ACF (or I am just missing something obvious 😄) - I can't find a version so far that double encodes the URL. I have tried passing URLs with %40 and %3D in them into cfhttp in various ACF2016 versions and I can't get it to request %2540 and %253D. Would it be possible to know the exact 2016 version you are on? Also just to confirm, your original post mentioned api.cfc:20 - did you mean coldfusion.cfc:20? And thank you for the kind words!

[doodi-v1]

Server Details

Server Product | ColdFusion 2016 Version | 2016.0.10.314028 Tomcat Version | 8.5.32.0 Edition | Developer Operating System | Windows 10 OS Version | 10.0 Update Level | C:/ColdFusion2016/cfusion/lib/updates/chf20160010.jar Adobe Driver Version | 5.1.4 (Build 0001)

I'm not 100% sure, but I'm guessing the latest 2018 version will act the same. I have yet to find any discrepancies between the latest 2016 and latest 2018 on functionality like this (cfhttp and url encoding).

And yes, you are correct, coldfusion.cfc:20, not api.cfc.

[jcberquist]

Thank you for all of the information, it has been very helpful. I spent a little while digging into this today, and now I think I know what is going on. There is a difference between S3 and other services with regard to Signature V4 request signing, specifically with regard to how a canonical request is constructed. According to the docs:

Normalize URI paths according to RFC 3986. Remove redundant and relative path components. Each path segment must be URI-encoded twice (except for Amazon S3 which only gets URI-encoded once).

The key is that "each path segment must be URI-encoded twice". Now in most cases that won't make a difference, but it does with your example URL above - and I couldn't reproduce with S3 URLs since they behave differently. But today I could reproduce it using the API Gateway with both ACF and Lucee - the cfhttp requests are behaving correctly; the issue is that I am not constructing non S3 canonical requests correctly.

I really appreciate you raising the issue!

[doodi-v1]

Very strange. I'm actually surprised AWS didn't catch the discrepancy and correct it on their end. Sounds like they need a case opened. I'm in awe at the level you went to track down the root cause. Need a job? ;-)

[jcberquist]

Alright, I have pushed v1.7.1 that includes the above commit. I believe it will fix this issue. Please let me know!

[doodi-v1]

Quick test passed. My demo app only tests a call to s3.listBucket and an execute-api call and both worked as expected. Thank you.

[jcberquist]

Great, glad to hear it. I had to switch from using URLEncodedFormat() to encodeForURL() due to a recent issue introduced (it makes double encoding using URLEncodedFormat impossible): CF-4204127 CF-4204045 With that change, however, things do seem to be working properly at the moment.

[jcberquist]

As a heads up, I have now published v1.7.2 to deal with the fact that encodeForURL encodes spaces as +'s and that isn't valid for the non query string part of a URL.

[doodi-v1]

My same tests were successful - but they don't specifically test the + issue, just that it didn't break the previous fix.

[jcberquist]

Yep, I get that, just wanted to make sure you knew I had pushed a further patch.