[x] Have you provided a code sample to reproduce the issue?
[x] Have you tested with the latest release to confirm the issue still exists?
[x] Have you searched for related issues/PRs?
[x] What's the actual output vs expected output?
Description
We are using this library to authorize our requests to the mastercard Secure Card on File API.
Thank you for this library, guys. It is small and neat, which is perfect to be embeddable in our secure environments.
But we had experienced issues while performing GET request with query parameters in it, since some of our query parameters' values contain reserved URL characters e.g. serivceId=Foo#Bar.
When supplying such a parameter in Mastercard's API and applying, of course, OAuth1 authorization, Mastercards' backend complains our OAuth1 base string is invalid. By comparing base string evaluated by oauth1-signer-python with the backends' one, we can see what differences occur in positions with reserved characters.
To Reproduce
Put print(base_string) statement inside of OAuth.get_oauth_parameters methods right after base string evaluation
Form any request with, for example, character # in URI. Like http://..../XXX%23YYY/ or https://.../?foo=b%23ar
Apply OAuth1 signature performed by oauth1-signer-python.
Send it to mastercard's backend (or any backend with rfc5849 OAuth1 validator)
Actual behavior
oauth1-signer-python base string (a slice)
%26serviceId%3DMyService%23Identifier%26
mastercard's SCF backend base string (a slice from backend's error message)
%26serviceId%3DMyService%2523Identifier%26
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
My platform is Ubuntu 18 with Python 3.8 on the board, but for me it seems like issue is platform independent.
Related issues/PRs
Has a similar issue/PR been reported/opened before?
Suggest a fix/enhancement
My suggestion is to apply rfc3986 percent encoding algorithm in rfc5849-compliant way:
unreserved characters MUST NOT be encoded
other characters MUST be encoded with percent encoding
Also we have to keep out environments' specifics (Python) in mind.
(ALPHA, DIGIT, "-", ".", "_", "~") MUST NOT be encoded
All other characters MUST be encoded
This is where the first incompliance at. This line especially https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L72 considers / to be safe while according to rfc5849 it MUST be encoded.
So taking for example foo=b/r when rfc5849-compliant parameters normalization procedure produces foo=b%2Fr, oauth1-signer procedure produces foo=b/r, which is incorrect.
After hacking oauth1-signer in order to force it to produce correct base string (according to my understanding of specs), I want to validate my idea in other people code in the Internet.
I found this library
https://github.com/oauthlib/oauthlib
This is another library which focuses on OAuth1/OAuth2 protocols. Although it's larger so it is harder for us to adopt this one and embed it in our secure environments.
But still, let's look how they perform percent encoding procedure:
https://github.com/oauthlib/oauthlib/blob/master/oauthlib/oauth1/rfc5849/utils.py#L53
Aha! This one perfectly matches to my idea about rfc5849 encoding
Demo playground
This is the comparison between oauth1-signer and oauthlib. oauthlibs' output matches my idea about rfc5849 and provides correct base string in order to authorize in Mastercards' API
https://repl.it/@huumanoid/OAuth1SignerPythonBaseStrings
Bug Report Checklist
Description We are using this library to authorize our requests to the mastercard Secure Card on File API.
Thank you for this library, guys. It is small and neat, which is perfect to be embeddable in our secure environments.
But we had experienced issues while performing
GETrequest with query parameters in it, since some of our query parameters' values containreservedURL characters e.g.serivceId=Foo#Bar. When supplying such a parameter in Mastercard's API and applying, of course, OAuth1 authorization, Mastercards' backend complains our OAuth1 base string is invalid. By comparing base string evaluated byoauth1-signer-pythonwith the backends' one, we can see what differences occur in positions withreservedcharacters.To Reproduce
print(base_string)statement inside ofOAuth.get_oauth_parametersmethods right after base string evaluation#in URI. Likehttp://..../XXX%23YYY/orhttps://.../?foo=b%23aroauth1-signer-python.Actual behavior oauth1-signer-python base string (a slice)
mastercard's SCF backend base string (a slice from backend's error message)
Screenshots If applicable, add screenshots to help explain your problem.
Additional context My platform is Ubuntu 18 with Python 3.8 on the board, but for me it seems like issue is platform independent.
Related issues/PRs Has a similar issue/PR been reported/opened before?
Suggest a fix/enhancement My suggestion is to apply rfc3986 percent encoding algorithm in rfc5849-compliant way:
Details
After studying https://tools.ietf.org/html/rfc5849 I realized that oauth1-signer-python implementation is not rfc5849-compliant. These two pieces of code are the source of incompliance https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L65 https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L100
First incompliance
https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L65 Let's discuss this first one, which is related to parameters normalization procedure Take a closer look at https://tools.ietf.org/html/rfc5849#section-3.4.1.3 describing request parameters transmission Taking about query parameters:
(ALPHA, DIGIT, "-", ".", "_", "~")MUST NOT be encodedThis is where the first incompliance at. This line especially https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L72 considers
/to besafewhile according to rfc5849 it MUST be encoded. So taking for examplefoo=b/rwhen rfc5849-compliant parameters normalization procedure producesfoo=b%2Fr, oauth1-signer procedure producesfoo=b/r, which is incorrect.Second incompliance
https://github.com/Mastercard/oauth1-signer-python/blob/master/oauth1/coreutils.py#L100 This piece of code is used generally to perform rfc3986 encoding of URI, method, normalized params, oauth parameters. And this is actualy the reason we faced issues with Mastercards' API. Somehow this function considers
%to be safe character, whereas it MUST be encoded according to https://tools.ietf.org/html/rfc5849#section-3.6.Other solutions across the Internet
After hacking oauth1-signer in order to force it to produce correct base string (according to my understanding of specs), I want to validate my idea in other people code in the Internet. I found this library https://github.com/oauthlib/oauthlib This is another library which focuses on OAuth1/OAuth2 protocols. Although it's larger so it is harder for us to adopt this one and embed it in our secure environments. But still, let's look how they perform percent encoding procedure: https://github.com/oauthlib/oauthlib/blob/master/oauthlib/oauth1/rfc5849/utils.py#L53
Aha! This one perfectly matches to my idea about rfc5849 encoding
Demo playground
This is the comparison between oauth1-signer and oauthlib. oauthlibs' output matches my idea about rfc5849 and provides correct base string in order to authorize in Mastercards' API https://repl.it/@huumanoid/OAuth1SignerPythonBaseStrings