Default value of parameter `bgresponse` has not accepted.

max-rocket-internet commented 4 years ago

Some of us are getting this message:

Google Password: 
Default value of parameter `bgresponse` has not accepted.
                Please visit login URL https://accounts.google.com/o/saml2/initsso?idpid=C00pn5t3o&spid=112173148568&forceauthn=false, open the web inspector and execute document.bg.invoke() in the console.
                Then, set --bg-response to the function output.

But running that in the JS console is not possible, it gives an error.

Are we supposed to run that command at the Google account selector screen? Or another point?
Is it possible to get this value from the source code of the page?

max-rocket-internet commented 4 years ago

OK, to be clearer, we need to visit the URL in a browser session with no credentials already configured, then you arrive at login screen (and NOT a account selector screen).

Then you can run document.bg.invoke(), but for us it still doesn't work, we get the same error again.

martinzugnoni commented 4 years ago

Same happening to me now after a fresh install of my Mac. Even though is still working for my colleagues in the office.

The POST request to https://accounts.google.com/signin/challenge/sl/password returns this error page:

Which ends up in the error described by @max-rocket-internet above here: https://github.com/cevoaustralia/aws-google-auth/blob/master/aws_google_auth/google.py#L258

It's weird because my browser is a last-version Chrome with JS enabled. I'm afraid with "app" it refers to iterm2, which I'm using to execute aws-google-auth.

max-rocket-internet commented 4 years ago

@martinzugnoni so you're getting that error message, even in a browser?

max-rocket-internet commented 4 years ago

The user-agent is set here: https://github.com/cevoaustralia/aws-google-auth/blob/master/aws_google_auth/google.py#L174

Perhaps that's the issue? Still strange how it's only a few people who have this problem.

martinzugnoni commented 4 years ago

No, login in the Browser works fine. It's only failing when I try to login from iterm2 using aws-google-auth and the Google SSO (Single Sing-on).

I'm using aws-google-auth==0.0.33, which was released on Nov 9th.

Indeed, it's strange that works for some people and fails for others. In my case, I'm the only one getting this error.

PanSotnik commented 4 years ago

I have had the same issue on two different computers, so perhaps specific to the account. Prior to upgrading, I was getting invalid SAML response errors.

Same here. Latest version throws 'bgresponse' error. Downgrading returns

Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.

metersk commented 4 years ago

Also, getting the exact same results as @PanSotnik

martinzugnoni commented 4 years ago

I just checked in my old computer, which was working under the exact same procedure during months, and it's failing with the same error now too. As @tinder-robolague said, it's probably something related to the account and not to the device.

martinzugnoni commented 4 years ago

More evidence proving the same theory. I went to my colleague work station, and:

He tried logging in using aws-google-auth with his account, and it worked fine
In the same computer, I tried to log in using my credentials, and it failed with the same error described above.

It's clearly something wrong with my account.

In the last few days I changed my password and changed the 2FA device in my Google account. Could that be the reason of the issue? Did any of you guys change any of password or 2FA settings lately?

PanSotnik commented 4 years ago

In the last few days I changed my password and changed the 2FA device in my Google account. Could that be the reason of the issue? Did any of you guys change any of password or 2FA settings lately?

I hadn't changed any of that and am still receiving the problem

metersk commented 4 years ago

Someone at my office hypothesized that this is happening for some, but not all, because Google is rolling this change out to user accounts in waves. 🤷‍♂

kyletitus commented 4 years ago

Just wanted to chime in and say at least one of our employees is running into this same issue since Monday morning. She cannot login via aws-google-auth on any work station. All other employees can login on theirs and hers. I was leaning towards it being a Google Account change and feel more confident in saying that it is after finding this issue.

hackerrdave commented 4 years ago

To follow up on this issue, I printed out the html response during the error scenario:

Couldn't sign you in. The browser you're using doesn't support JavaScript, or has JavaScript turned off.</p><p>To keep your Google Account secure, try signing in on a browser that has JavaScript turned on

brokenthumbs commented 4 years ago

Is there a way to update the code so that bgresponse is not js_disabled? My guess is that Google will start blocking requests that have js_disabled in the headers.

EDIT: I didn't read the old merged PRs. Looked over the merged PRs and found this: https://github.com/cevoaustralia/aws-google-auth/pull/150

In reference to https://github.com/cevoaustralia/aws-google-auth/issues/160#issuecomment-566545053, was the bgresponse passed in with --bg-response? Something must be weird with Google, because I was able to replicate, execute document.bg.invoke() in a logged out incognito browser, take the value, and pass that value as a string to --bg-response, and re-invoking aws-google-auth

Seems pretty cumbersome. And there's no guarantee on how long that value lasts for either. Any ideas for a longer term solution? I know that the heaviest solution might be the headless browser, as opposed to requests library, as suggested by @ruimarinho

EDIT: A coworker of mine tried this same approach, but also encountered the same error as https://github.com/cevoaustralia/aws-google-auth/issues/160#issuecomment-566545053. So, this approach seems to be inconsistent, possibly? Unsure why getting the value for bgresponse would work for one Google account, but not a different Google account.

EDIT: Is this a possible workaround, somehow? https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-saml.html https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-web-identity.html

ruimarinho commented 4 years ago

Hi everyone. I am the author of the PR that allows modifying bgresponse.

Google seems to have introduced new measures to perform liveness checks on the authentication page based on criteria I don't fully understand yet. It could be related to their new Context Aware feature they're rolling out to G Suite customers but who knows. Like many other teams, ours is experiencing this in different waves, but our conclusion is that once you enter this 'funnel', there is no way out.

For background, after some investigation, I found out that generating a bgresponse token on a credible browser would allow bypassing this restriction. This is what led to the first version of the PR which, although fragile, was working fine until today.

I believe Google has now blacklisted this approach and therefore fetching the token dynamically would not solve this problem. I think our next attempt needs to have a browser context. Something like pyppeteer would be a good candidate for this if anyone would like to give it a try, but it's really a question of time until they detect that as well. May be worth it if it works for the next 8-12 months.

In the mean time, assumeRoleWithSAML works fine if you're ok with leaving your terminal. In my case, that's a big no, but until we can find an alternative, there's an extension for Chromium-based browsed to do that which works fine (prolane/samltoawsstskeys). I suggest you download and install it manually to avoid potential security issues.

brokenthumbs commented 4 years ago

@ruimarinho

Would it be possible to automate the process in retrieving the --saml-assertion? Based on this AWS Post, the process for viewing the SAML Response would be using the Developer Tools, logging the SAML Response, and using that as the --saml-assertion.

Would it be possible to use something like pyppeteer to programatically get this SAML Response? Or would Google somehow flag this, and not allow you to get a SAML Response? I also assume you would have to be already logged in on your browser for this approach to work.

ruimarinho commented 4 years ago

At this point I have no hope. Tried to prototype something with puppeteer and hit the same wall. Tried the stealth plugin as well but that didn't work. See https://github.com/puppeteer/puppeteer/issues/4871 for background.

Trying to see if I can share state/sessions between an existing Chrome installation with a good login and to use that for automation puppeteer. I don't have high expectations though.

max-rocket-internet commented 4 years ago

At this point I have no hope

Don't lose hope: https://aws.amazon.com/about-aws/whats-new/2019/11/use-the-aws-cli-v2-preview-with-aws-single-sign-on-to-increase-developer-productivity/

Still can't believe it's been this long without a native AWS solution for this..

brokenthumbs commented 4 years ago

@max-rocket-internet

Still can't believe it's been this long without a native AWS solution for this..

max-rocket-internet commented 4 years ago

@brokenthumbs I meant without a proper solution for SSO with the AWS CLI, of which I think there's surely more than dozens of us 😅😅

brokenthumbs commented 4 years ago

After reading the AWS blog post more carefully, regarding aws2 configure sso, I realize that the new feature only supports retrieving credentials managed in AWS SSO or Microsoft Active Directory.

Here's hoping that AWS updates aws2 configure sso to handle Google SSO.

SpamapS commented 4 years ago

I wonder if this could be fixed simply by opening a browser window, instead of fetching directly.

pauldraper commented 4 years ago

Still can't believe it's been this long without a native AWS solution for this..

It's not really an AWS thing though.

It's a "use Google SSO from the command line" thing. That's just an impossible way to use Google SSO, apparently. Google bad.

brokenthumbs commented 4 years ago

@pauldraper

By way of aws sts assume-role-with-saml, AWS does support using Google SSO. There are a few extra steps with the Developer Tools and the SAML Response.

aws2 configure sso doesn't support using Google SSO. But, potentially AWS will somehow support a connection directly to Google SSO? They already support Microsoft Active Directory, so maybe there's some hope for Google SSO, or even Google GSuite LDAP.

Google is cracking down on robot interactions with the automated browser interaction. @ruimarinho has been experimenting with Pypetteer, so that might be a viable solution, until Google cracks down on that too.

futurekill commented 4 years ago

Glad I stumbled upon this issue. I configured my account to use SAML a few weeks ago using saml2aws and everything went fine. Earlier this week I tried helping a college get set up and it wasn't working inexplicably. Stupid Google.

ruimarinho commented 4 years ago

I have a working solution with puppeteer. I'll try to get it published for preview today or tomorrow. It's still fragile solution but nonetheless easier to setup than AWS SSO, assuming it's at all possible (I have tried it but without success yet).

pauldraper commented 4 years ago

easier to setup than AWS SSO, assuming it's at all possible

AWS SSO can be set up fine, but the AWS CLI integraton works in the same way that aws-google-auth does (HTTP requests to the IdP), and so would have to solve the same anti-robot issues as aws-google-auth for anyone using Google as their IdP.

I created a Chrome extension that makes it easy to retrieve the SAML response from a normal web browser login:

https://github.com/rivethealth/chrome-saml

And then a simplified aws-google-auth-clone that accepts SAML as an input:

https://github.com/rivethealth/aws-cli-saml

Pros:

Works for any SAML IdP.
Robust for any 2FA or U2F or VPN or geolocation or any other wacky requirements.
Saves cookies, thus possibly skipping repetitive 2FA steps.

Cons:

Requires leaving the command line. However, aws-google-auth + 2FA already required not only leaving the command line but accessing another computing device altogether, so no large loss I think.

Conceivably, aws-google-auth could be modified to accept the SAML responses as input like aws-saml, accepting this approach as an option.

max-rocket-internet commented 4 years ago

@pauldraper is this a completely different project and repo then? Or a PR for this one?

pauldraper commented 4 years ago

@max-rocket-internet, for ease of my development, they are both separate projects (chrome-saml is a browser extension; aws-cli-saml is a CLI tool similar to this project).

But it looks like @pmarket is adding a CLI option to aws-google-auth in #161, making this project a super-set of aws-cli-saml. With that PR, you can use chrome-saml in conjunction aws-google-auth --saml-assertion.

Again, at that point, there's nothing "Google/GSuite" about it, and you actually have a general-purpose SAML setup: works for Google, Active Directory, whatever.

mhenniges commented 4 years ago

For those seeing inconsistent results with the browser-based document.bg.invoke() workaround: that workaround worked fine for a few weeks, and then started failing. After much fiddling, it seemed to have something to do with cookies reflecting multiple google accounts being present in my browser session (personal account + work account).

After:

New incognito session
drop all cookies related to google
navigate to url provided by aws-google-auth
enter email address I want to auth with
execute document.bg.invoke()

the workaround then functioned again.

max-rocket-internet commented 4 years ago

For one of my colleagues the problem just went away 🤷‍♂️

lmtien commented 4 years ago

I had this problem for the last 2 weeks, and now it's gone also (just tried it). 🙋‍♂

martinzugnoni commented 4 years ago

Same for me! It's now working.

zeroaltitude commented 4 years ago

It didn't work for me unfortunately -- the bgresponse thing is still an issue. I used a combination of the idea of getting the SAML directly (from the above chrome-saml idea) and using Pyppeteer as an automation tool to launch the window (without requiring the user to manage a specific browser on a specific platform) and grab the SAML. The way it works is that it prompts the user for the username and password in GSuite, then automates the rest. The only slight ugliness is you have to disable automation and run headful in order for google not to stomp the login request. It is, however, somewhat more elegant than having to install another plugin and grab the saml manually. I'd be happy to open source this if anyone would find it useful.

ruimarinho commented 4 years ago

Took me a bit longer than expected as I did not want to force the end-user to manually relaunch the utility on first run.

Here is the promised alternative in puppeteer - called gsts (short for Google STS) if anyone would like to help testing:

Install the package via npm:

npm install --global gsts

or via yarn:

yarn global add gsts

In its current form, it's very simple and without much error handling yet. It covers only the most basic scenarios. Because of that, it works best with all environment variables it supports set.

It works very well for me but I have no idea how well it works for others yet nor for how long it will last.

SpamapS commented 4 years ago

Seems like whatever anti-bot thing Google did has been reverted or improved, and this issue seems resolved. I just tried it after a few weeks and it works again. Still seems like a good idea to go down the path of using the browser so this doesn't happen again.

metersk commented 4 years ago

Hey all,

Is anyone else still having issues with this? Seems like it was not fixed for me...

kyletitus commented 4 years ago

It has resurfaced as of last week. My previously affected users are broken again.

ruimarinho commented 4 years ago

@kyletitus still broken here. I am now using the gsts implementation mentioned above which has been working well since its first release.

nonspecialist commented 4 years ago

Hey folks, can you test 0.0.34 and see if it resolves your issues?

winjer commented 4 years ago

Can confirm, 0.0.34 has resolved the issue for me.

metersk commented 4 years ago

i previously had the bgresponse() error and upgrading to 0.0.34 gives me the following error:

ERROR:root:'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/Users/metersk/.virtualenvs/py3_new/lib/python3.7/site-packages/aws_google_auth/__init__.py", line 75, in cli
    process_auth(args, config)
  File "/Users/metersk/.virtualenvs/py3_new/lib/python3.7/site-packages/aws_google_auth/__init__.py", line 222, in process_auth
    google_client.do_login()
  File "/Users/metersk/.virtualenvs/py3_new/lib/python3.7/site-packages/aws_google_auth/google.py", line 235, in do_login
    passwd_challenge_url = 'https://accounts.google.com' + form.get('action')
AttributeError: 'NoneType' object has no attribute 'get'

majeska commented 4 years ago

03/01 Two days ago I was getting this: "ERROR:root:'NoneType' object has no attribute 'get' 03/03 Tried again, now I'm getting "Default value of parameter bgresponse has not accepted."

$ aws-google-auth -V $ aws-google-auth 0.0.34

bigbee commented 4 years ago

I also have had this issue since december. When I started using 0.0.33 it worked. Using 0.0.34 the problem is back again.

juanma-clavero commented 4 years ago

in case this helps anybody, I had this problem starting 2020-03-09. The instructions to go to the URL and get the document.bg.invoke() didn't solve the issue. I kept trying with different versions of the library with no luck several times on that day and on 2020-03-10. Suddenly, a retry on 2020-03-11 asked me to solve a captcha. With the captcha done, the bg-response wasn’t an issue anymore and the line google_config.bg_response = None at the end of my ~/.aws/config. So maybe we need a way to force aws-google-auth to request a captcha if the bg-response is not valid

jhaynie commented 4 years ago

this is happening again today and can't seem to get it to work. using 0.0.34

Zlender commented 4 years ago

I just encountered this for the first time after testing the tool for a week. I can't seem to trigger any captcha in the browser, bg_response to a value or None doesn't seem to work.

[Edit]

The browser you're using doesn't support JavaScript, or has JavaScript turned off.</p><p>To keep your Google Account secure, try signing in on a browser that has JavaScript turned on.

Is the error I'm getting it was working fine just a couple of days ago.

stevemac007 commented 4 years ago

The number of different paths through google SSO are wide and varied. I've not experienced this, and so hard to try and help troubleshoot - I can confirm the recent build 0.0.34 and 0.0.35 (just released) are working fine for me with multiple logins across different AWS accounts.

I did run into the captcha issue for a few weeks a while back (Jan timeframe) and found after I relogged in via a browser and reviewed the recent security history my issues went away.

I've found if you click the link in the email asking if you recently logged in and confirm it, the captcha and extra hoops seem to go away. But this might just be anecdotal .

stevemac007 commented 4 years ago

I've started #181 to debug all the HTML - will get this reviewed and merged and then we can at least capture all of the HTML traffic to see if there is something in the pages that are common.

stevemac007 commented 4 years ago

I also found out there is a mode in Google Security settings - Less secure apps & your Google Account - https://support.google.com/accounts/answer/6010255?hl=en-GB

It seems you can enable Less secure apps here - https://myaccount.google.com/lesssecureapps - I wonder if this is related.

cevoaustralia / aws-google-auth

Default value of parameter `bgresponse` has not accepted. #160