id-challenge HTML element does not always contain a JSON value

FabianFrank commented 5 years ago

Sometimes aws-google-auth fails with the following exception:

  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/__init__.py", line 197, in process_auth
    google_client.do_login()
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 259, in do_login
    sess = self.handle_sk(sess)
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 371, in handle_sk
    raise e
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 366, in handle_sk
    challenges = json.loads(challenges_txt)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I think the issue is because of the challenge HTML form having changed. During a failed run, it looks like this:

            <div class="LJtPoc" jsname="Ki8mld" jscontroller="qNel6e" jsaction="rcuQ6b:WYd">
                <form method="POST" id="challenge" action="/signin/challenge/sk/6" jsname="rzWj5" jscontroller="HNBfvc" jsaction="submit:zbvklb" jsshadow>
                    <content>
                        <input name="challengeId" type="hidden" id="challengeId" value="6">
                        <input name="challengeType" type="hidden" id="challengeType" value="2">
                        <input name="continue" type="hidden" value="https://accounts.google.com/o/saml2/initsso?idpid=REDACTED&amp;spid=REDACTED&amp;forceauthn=false&amp;from_login=1&amp;as=REDACTED">
                        <input name="scc" type="hidden" value="1">
                        <input name="sarp" type="hidden" value="1">
                        <input name="checkedDomains" type="hidden" value="youtube">
                        <input name="pstMsg" type="hidden" value="0">
                        <input name="TL" type="hidden" value="REDACTED">
                        <input type="hidden" name="gxf" id="gxf" value="REDACTED">
                        <div jsname="KrwUDc"><img jsname="TqVmm" class="JC07Dd" src="//ssl.gstatic.com/accounts/marc/gnubby_icon.png" alt="">
                            <div jsname="kwHene"></div>
                            <div class="EGmPD" jsname="BCqkPb">Insert your security key</div>
                            <div class="VnJmLc" jsname="NhJ5Dd">If your security key has a button, tap it.
                                <br>If it doesn't, remove and re-insert it.</div>
                            <div>
                                <input type="hidden" name="id-challenge" jsname="wCVnAe" value="REDACTED_STRING_NOT_JSON">
                                <input type="hidden" name="id-assertion" jsname="n6knUb">
                            </div>
                            <div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;REDACTED&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>
                            <div class="ARshqb">
                                <input type="checkbox" name="TrustDevice" id="trustDevice" class="aCOJmf" checked><span>Don&#39;t ask again on this computer</span>
                                <div class="Bfmfyc" role="tooltip">
                                    <div class="x7qQqf"></div>
                                    <div class="hzC8Lb">For your convenience, keep this checked. On shared devices, additional precautions are recommended. <a href="https://support.google.com/accounts/?p=securesignin&amp;hl=en" target="_blank">Learn more</a></div>
                                </div>
                            </div>
                        </div>
                    </content>
                </form>
            </div>

Based on this it looks like it comes down to the value string of id-challenge no longer being a stringified JSON object, but instead a "random" string that looks like "...hUTYdnvUG6M25UzFVz...". I poked around but it wasn't clear to me what the right way is to fix this, especially because

<div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;fabian@ruist.com&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>

looks like it has the relevant data but also looks like it isn't simply JSON...

This has started happening approximately 5 days ago.

In case it is relevant, saving the html and opening it in a browser (Chrome) renders this page:

jaydgoss commented 5 years ago

This is happening to our org as well. aws-google-auth has become unusable for accounts that face CAPTCHA challenges.

adcreare commented 5 years ago

We are seeing this as well what is odd is just rerunning the command multiple times eventually returns a page that does have valid json contained within. I guess eventually it gets an old page in the google fleet or falls back to an older version of the page?

FabianFrank commented 5 years ago

I'm experiencing the same behavior like @adcreare, it fixes itself after a few reruns. I'd be happy to try and make a PR for this, but I don't know enough SAML to find the values that are needed in the new Google page.

waeltken commented 5 years ago

For me it has completely stopped working now... Yesterday I noticed this issue for the first time.

olitheolix commented 5 years ago

Same here. I got the occasional error in the past but a re-run fixed it. Since this morning it has stopped working altogether for me.

funollet commented 5 years ago

Same here, stopped completely.

f-barth commented 5 years ago

Same here => previously one or two retries worked to get a "valid JSON" version. Since yesterday it stopped working.

olitheolix commented 5 years ago

I just found out that this only affects the code path for Yubi key authentication, at least for me.

As a (temporary) workaround, I removed the keys from my Google account and use Google's Authenticator app instead - not ideal, but at least I can authenticate again.

dcherniv commented 5 years ago

@olitheolix thanks for this workaround. I can reproduce with yubikey only as well. Falling back to google authenticator works for me as well.

sinkr commented 5 years ago

I, too, am having the same issue with a YubiKey.

FabianFrank commented 5 years ago

It is now consistently broken for everyone on my team as well, including me. Disabling yubikey fixes it as well. It seems like the issue is that Google does some kind of browser detection and if it's not Chrome they want you to prompt to install Chrome. I've tried hardcoding the user agent in google.py to Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36, but didn't see a change in behavior.

dinoboy197 commented 5 years ago

This is also broken for me. I also use a Yubikey.

maxleonca commented 5 years ago

same here, yubikey as well.

nbetm commented 5 years ago

same here, I had to remove the yubikey and use google prompt as default.

mikamikuh commented 5 years ago

Same here. Looks like there is no way to change the default 2FA method on Google account setting, so you'll have to remove the yubikey to use Google's Authenticator app for workaround.

djk commented 5 years ago

Also having this issue with U2F (Yubikey).

bryan4tw commented 5 years ago

Just to be clear, this doesn't appear to be an issue with Yubikey, but with the U2F process. My Google TItan keys also have the same problem.

brainstorm commented 5 years ago

To anybody in this issue (or the author of the tool (@stevemac007 ?)): can you recognize what's the payload of the id-challenge field when it's not JSON? Is it an just AES-encrypted message (after base64-decoding)? Something else?

I'm saying this because perhaps we could just detect when it's not a JSON and carry on with the auth (replying to the challenges using the Yubikey/Titan keys output)?:

https://github.com/cevoaustralia/aws-google-auth/blob/master/aws_google_auth/google.py#L363

/cc @reisingerf

marksteele commented 5 years ago

Same issue, with yubikey

vlesierse commented 5 years ago

Same issue here as well....

stevemac007 commented 5 years ago

It looks repeatable enough that we should be able to work something out.

The challenge with this whole tool is that it is reverse engineered from behavior in the wild, and google can (and has) change the format of these pages as they see fit.

It looks like this is another one of those cases.

If someone with access to one of the hardware keys that are having problems has the capability to help troubleshoot and reverse-engineer what is happening, I'd willingly accept (and help with) updates to the tool.

I don't have one of these keys at the moment, so not something I can simulate, and hence resolve.

If you have ways in which I can replicate this on my account I'm willing to take on the task of trying to work out a way around this.

Just note, we can't promise anything - there have been a few things - like Captcha problems of the past - that required considerable time to try and work out, and as quickly as they were issues they seemed to go away again. There is no real view into the complexity that is the Google SSO machine from the outside.

sinkr commented 5 years ago

Set up some kind of campaign / funding avenue and I'm sure a bunch of us will have no trouble donating the money required to buy a YubiKey.

davidrjonas commented 5 years ago

On Mar 28, 2019, at 3:26 PM, Robert Sink notifications@github.com wrote:

Set up some kind of campaign / funding avenue and I'm sure a bunch of us will have no trouble donating the money required to buy a YubiKey.

If https://www.yubico.com/product/security-key-by-yubico/#security-key will work and you think you can do it, make an amazon wishlist with it and I’ll buy it for you right now. $20 is definitely worth it to me.

brainstorm commented 5 years ago

This change on the Google U2F reference implementation might explain things:

https://github.com/google/u2f-ref-code/commit/d8998ff86c1b75fb3856cbf5771734df21377402

TL;DR: Deprecating browser plugin side-loading in favor of WebAuthN.

/cc @wandergeek

philstevenson commented 5 years ago

Think I am going to switch to this to circumvent this issue: https://github.com/flowcommerce/aws-credentials-broker

sinkr commented 5 years ago

Hmm, uninstalled, switched to a virtual MFA and am still getting the JSON errors. Any ideas?

hadalin commented 5 years ago

I'm also having this issue with YubiKey (U2F). What worked for me was using Google Prompt for 2FA instead of YubiKey.

Colgaton commented 5 years ago

Maybe there is a way to call other 2fa methods instead of yubikey? In my google account yubikey is the default method, with google prompt and authenticator coming in second and third places.

adcreare commented 5 years ago

Alright time this got fixed :) I've created a PR https://github.com/cevoaustralia/aws-google-auth/pull/136 that fixes it for my account using a single standard yubikey on my google corp account. Very keen to hear from others if this works for them as well.

I'm also no python expert so if you spot anything horrible, let me know and I'll fix it

stevemac007 commented 4 years ago

This looks to be resolved with the 0.0.34 release.

geft commented 4 years ago

No, I'm still getting it on 0.0.35. Not sure of the reason however.

ERROR:root:'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/home/geft/.aws-google-auth/local/lib/python2.7/site-packages/aws_google_auth/__init__.py", line 76, in cli
    process_auth(args, config)
  File "/home/geft/.aws-google-auth/local/lib/python2.7/site-packages/aws_google_auth/__init__.py", line 233, in process_auth
    google_client.do_login()
  File "/home/geft/.aws-google-auth/local/lib/python2.7/site-packages/aws_google_auth/google.py", line 235, in do_login
    passwd_challenge_url = 'https://accounts.google.com' + form.get('action')
AttributeError: 'NoneType' object has no attribute 'get'
geft@ubuntu:~/.awsudo$ aws-google-auth --version
aws-google-auth 0.0.35

cevoaustralia / aws-google-auth

id-challenge HTML element does not always contain a JSON value #128