search

cevoaustralia / aws-google-auth

Provides AWS STS credentials based on Google Apps SAML SSO auth (what a jumble!)
MIT License
538 stars 180 forks source link

Resolve issue 128 - yubikey auth flow broken by google page update #136

Closed adcreare closed 5 years ago

adcreare commented 5 years ago

This makes an attempt to resolve issue https://github.com/cevoaustralia/aws-google-auth/issues/128 Works for me with a single yubikey on my google corporate account.

Google changed the id-challenge field to contain just the challenge txt, they include the other information now hidden away inside a div tag further down in a json like object that contains a bunch of nulls. They also provide some of the data in base64 now but their api expects base64 url encoded on some of the fields, despite all that being base64'd itself.

This PR has resolved the issue for us, I'd be very keen to hear from others if this works for them. Comments, feedback etc welcome, I'm not exactly a python expert so any points there welcome too :-)

coveralls commented 5 years ago

Coverage Status

Coverage decreased (-1.6%) to 46.756% when pulling aa4ab7615544663ed15de33a9d482d1bfc5689e8 on adcreare:master into 6f5919d0408f117a6fa0ebc61ee23e5559ad39d5 on cevoaustralia:master.

hadalin commented 5 years ago

Seems to be working for me 👍

You should probably add a test.

mhenniges commented 5 years ago

For some reason, this doesnt work for me with a yubikey 4, but I'm not sure exactly why yet, based on my limited knowledge it looks like the right fields are being sent. I'll try some more testing later and try spot the issue.

adcreare commented 5 years ago

@hadalin agreed re tests: I'd like to expand the tests a lot if I can so its easier in the future for others to understand the flow as well - took me awhile to get my head around what was going on

@mhenniges interesting - do you get an error of any kind?

I'll try update the tests later in the week

mhenniges commented 5 years ago

@adcreare , frustratingly, no, I don't get any error other than the final 'Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.' saml.html indeed does not have a saml response, its that same html page that you were seeing before this PR.

I'm not exactly sure what to look for to make additional progress on this. So far, I've done the following: 1 - added logging for pretty much every variable and http response in order to check that things are getting parsed and identified properly. I think that part is working fine; all the values look sane to me. 2 - added a print of json.dumps(u2f_challenges) in the handle_sk method in google.py; and validated that the printed json does successfully result in a signed challenge response when fed into ' u2f-authenticate https://accounts.google.com'. I did patch my u2f_host library to allow that facet, as the comments in u2f.py discuss.

3 - dropped the starting url (https://accounts.google.com/o/saml2/initsso...) into my browser, and captured a trace as I went through the flow. Everything worked fine in chrome, and I ended up on the AWS console. I then captured a trace of aws-google-auth trying to authenticate and compared the two.

In the post to submit the u2f challenge step, I see the following differences:

Present in the form data of browser trace but absent in aws-google-auth trace: checkConnection: youtube:285:1 flowName: GlifWebSignIn

Fields gfx, TL, id-challenge, and id-assertion had different contents. Also, in the id-assertion, there was an additional field in the browser version: "sessionId":""

In the 'continue' field, the 'as' term was different, but in each it seemed consistent with the redirects that had come previously.

Also, the browser trace has a cookie called 'GAPS'.

Of course those differences may be completely normal...

I'm happy to do other tests if there's something specific I should try. I don't have any leads at all at the moment.

sinkr commented 5 years ago

Hmm, this does not work for me. I have verified that I have installed your PR and am invoking it, however, I still get this error after captcha verification (you'll notice I incremented the version to 0.0.32 just so I could verify I'm running the correct code):

ERROR:root:Object of type bytes is not JSON serializable Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/__init__.py", line 72, in cli process_auth(args, config) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/__init__.py", line 212, in process_auth google_client.do_login() File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/google.py", line 293, in do_login sess = self.handle_sk(sess) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/google.py", line 417, in handle_sk auth_response_dict = u2f.u2f_auth(u2f_challenges, facet) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/u2f.py", line 61, in u2f_auth return u2f.authenticate(device, json.dumps(challenge), File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type bytes is not JSON serializable

adcreare commented 5 years ago

very odd - i'm using it daily and it seems ok. We have a whole team who use it as well so I'll push this fix out to them and see how they go.

out of interest what version of python are you guys running, i'm running 2.7

sinkr commented 5 years ago

very odd - i'm using it daily and it seems ok. We have a whole team who use it as well so I'll push this fix out to them and see how they go.

out of interest what version of python are you guys running, i'm running 2.7

Interestingly enough, 3.7. I reinstalled it with 2.7 and don't get the above error, but I now get this which I see @mhenniges got above:

Touch the flashing U2F device to authenticate... Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.

mhenniges commented 5 years ago

@adcreare - I'm running 2.7.16 as installed by brew on OSX 10.14.5, and my dependencies resolved as follows:

$ pipdeptree -p aws-google-auth,python-u2flib-host aws-google-auth==0.0.31

python-u2flib-host==3.0.3

how does this compare to your setup?

adcreare commented 5 years ago

I think i found the issue for the SAML not found - basically google wasn't always accepting the auth response when it includes base64 padding in the challenge response.

I've added something to remove the padding before its passed to the u2f code to sign and that seems to fix it.

base64 padding shouldn't (in theory) cause an issue but seems that it is in this case. Give it a try see if it helps. I haven't had a chance to look into the python 3.x issue - I'll do that when I get a chance.

mhenniges commented 5 years ago

@adcreare nice work! I've got nothing but successes with the current version. Is this ready to merge then @stevemac007 ? Anything I can do to help push it over the finish line if not?

robertsosinski commented 5 years ago

Hey all,

So I got it almost working with pulling @adcreare master branch on Python 2.7, however when I get to the Yubikey part, I get this:

ERROR:root:No U2F device found. 5 attempts remaining Insert your U2F device and press enter to try again...

No matter how many times I use my Yubikey, and I see the input streaming into the command prompt, it fails.

I'm using a Yubikey 5 NFC.

Thanks!

robertsosinski commented 5 years ago

Just wanted to chime in again. Confirmed this same issue on both Windows 10 cmd, and Ubuntu 18.04 Linux Subsystem on Windows 10.

Thanks!

adcreare commented 5 years ago

Just wanted to chime in again. Confirmed this same issue on both Windows 10 cmd, and Ubuntu 18.04 Linux Subsystem on Windows 10.

Thanks!

Haven't changed any of the U2F device detection stuff - I'd suggest making sure you have install the package with U2F support. We have had this issue internally on some macs our solution was to install https://www.python.org/ftp/python/2.7.15/python-2.7.15-macosx10.9.pkg and that cleared it up.

adcreare commented 5 years ago

@adcreare nice work! I've got nothing but successes with the current version. Is this ready to merge then @stevemac007 ? Anything I can do to help push it over the finish line if not?

👍 awesome! That's great news - I could do with some help from someone with better python skills to make this work on 2.7x and 3.x python. This branch 3.x doesn't work atm - due to string and byte changes in 3.x. I'll try and get it sorted out but if anyone is an expert and can make any suggestions, please do! :)

robertsosinski commented 5 years ago

Hi again,

So I installed from source with pip install -e .[u2f], is there anything else I should try?

Am I doing something else wrong? After it asks for the Yubikey, I just put it in and press the glowing Y symbol (where I see the input stream in) and it ends up asking again that no U2F key is found.

I've tried both Python 2.7.16 and 2.7.15.

Thanks for all your help!

-Robert

sinkr commented 5 years ago

@adcreare: [insert diety here] bless you!

I was finally able to get this to work by forcibly using Python 2.7!

mhenniges commented 5 years ago

@adcreare I did some hacking at this and with small changes I'm getting good results for both 2.7 and 3.7. I'm not sure what's the most convenient way to share those, but here's a diff:

--- a/aws_google_auth/google.py
+++ b/aws_google_auth/google.py
@@ -138,7 +138,10 @@ class Google:
         typeOfInput = type(input)
         if typeOfInput == dict:  # parse down a dict
             for item in input:
-                return Google.find_key_handle(input[item], challengeTxt)
+                rvalue = Google.find_key_handle(input[item], challengeTxt)
+                if rvalue is not None:
+                    return rvalue
+
         elif typeOfInput == list:  # looks like we've hit an array - iterate it
             array = list(filter(None, input))  # remove any None type objects from the array
             for item in array:
@@ -408,9 +411,10 @@ class Google:

         # txt sent for signing needs to be base64 url encode
         # we also have to remove any base64 padding because including including it will prevent google accepting the auth response
-        challenges_txt_encode_pad_removed = base64.urlsafe_b64encode(base64.b64decode(challenges_txt)).strip('=')
+        challenges_txt_encode_pad_removed = base64.urlsafe_b64encode(base64.b64decode(challenges_txt)).strip('='.encode())
+
         u2f_challenges = []
-        u2f_challenges.append({'version': 'U2F_V2', 'challenge': challenges_txt_encode_pad_removed, 'appId': appId, 'keyHandle': keyHandle})
+        u2f_challenges.append({'version': 'U2F_V2', 'challenge': challenges_txt_encode_pad_removed.decode(), 'appId': appId, 'keyHandle': keyHandle.decode()})

         # Prompt the user up to attempts_remaining times to insert their U2F device.
         attempts_remaining = 5
adcreare commented 5 years ago

@mhenniges that looks wonderful! I'll apply this diff tomorrow and update the PR.

adcreare commented 5 years ago

Updated to make python 3 work. Thanks again @mhenniges - ready for a review/merge

mhenniges commented 5 years ago

@stevemac007 - Anything else we can do to help here so this can merge in?

FabianFrank commented 5 years ago

I can confirm that with this PR merged login with a yubikey is working again.

jtatum commented 5 years ago

This resolved the error message for me, but on a system with no security key, auth now fails. I expected the --disable-u2f option to allow me to select some alternate auth method but it doesn't - I still get prompted to insert my key.

ahilsend commented 5 years ago

This fix solves the Yubikey auth for me.

lyonsy commented 5 years ago

This also fixed for me. Using this: https://www.yubico.com/product/yubikey-5-nfc

neilramsay commented 5 years ago

Release v0.0.32 works for my Yubikeys (firmware versions 4.3.5, and 5.1.1). Thanks :-)