search

gaubert / gmvault

gmail backup software
http://www.gmvault.org
GNU Affero General Public License v3.0
3.58k stars 288 forks source link

Google will revoke Gmvault access March 31, 2019 #335

Closed matthewhelmke closed 5 years ago

matthewhelmke commented 5 years ago

FYI, I just received this email. I've been using Gmvault for a few years assuming it was abandoned, since there have been no commits since 2016, but thought I would create this issue anyway, just in case...

Hello,

Although you don’t need to take any action, we wanted to let you know that the following third-party apps will no longer be able to access some data in your Google Account, including your Gmail content. This change will go into effect starting March 31, 2019.

Gmvault

We are making this change as part of ongoing efforts to make sure your data is protected and private. These apps haven’t yet complied with our updated data privacy requirements announced on October 8, 2018.

You can always view, manage and remove apps you’ve given access to your account by visiting your Google Account.

Thanks, The Google Accounts team

LufoX11 commented 4 years ago

I'm using G Suite. All you have to do is enable less secure apps access: https://myaccount.google.com/lesssecureapps

Then just use -p option

ikyriakidis commented 4 years ago

FYI, it seems google is pretty sticky with handing out the API access, so it may take some time to get this working.

Screenshot 2019-03-22 at 00 23 00 Screenshot 2019-03-22 at 00 32 13

I am having a similar problem and the "Submit for verification" is only active if i enter the following \

Any ideas?

lukaszpiwko commented 4 years ago

I'm writing this in case anyone is wondering if Gmvault is still working. I have just performed auth credentials update as per https://github.com/gaubert/gmvault/issues/335#issuecomment-475437988 and all emails are syncing without issue.

Thanks again for this great tool!

mimijojo commented 3 years ago

Procedure to set-up works fine :

#

Procedure to get yourself a new client ID & secret:

* Start here: https://console.developers.google.com

* Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)

* Create a new project (at the top); of choose one you already have, if that makes sense

* Go to https://console.developers.google.com/apis ; click `+ ENABLED APIS AND SERVICES` at the top

* Search for Gmail; select it; click `Enable`

* Go to https://console.developers.google.com/apis/credentials; choose `OAuth Consent Screen` tab

* Fill name (enter anything), click `Add scope`

* Select the checkbox on the `https://mail.google.com/` line; click `Add`

* Click `Save`

* Click `Create Credentials`; choose `OAuth client ID`

* Application Type: `Other`; give it a name (anything; I suggest `gmvault`, since this credential will be used by gmvault)

* Click `Create`

* You will get a `client ID` and `secret`. Those two values needs to replace the existing ones in `$HOME/.gmvault/gmvault_defaults.conf` : `gmvault_client_id=...` and `gmvault_client_secret=...`

* If you are running gmvault 1.9.1, make sure `conf_version=1.9` in that same conf file, and NOT `conf_version=1.9.1`, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change `conf_version` if you are running 1.9.2-beta-1 or higher.)

* Finally, obtain a new OAuth token using the following command:
  `gmvault check --renew-oauth2-tok your_email_address@gmail.com`
  Repeat this last step for all other Gmail accounts you are backing up using Gmvault on this particular computer, or the last 3 steps on other computers. (No need to create multiple clientIDs & secrets to backup multiple accounts or for multiple installs.)

But it stops working after a while (10 days or so, or maybe a number use, I've no clue.)

Can someone help on how to fix or troubleshoot this. I'm really stuck.

mimijojo commented 3 years ago

This works indeed, but it stops working after a while (10 days or so, or maybe a number use, I've no clue.) I get "http error 400 or 401". Anyone some expertise on this ? Or a way to troubleshoot ?

PS : In the meantime, current workaround is recreating the token each time again.

gboudreau commented 3 years ago

@mimijojo I updated my instructions above with more details, since the flow changed since I wrote that. Maybe re-read them now, to make sure you did the right thing when setting it up. I myself created a new project to test those updated instructions, and updated my OAuth tokens with that project. I know my last project allowed me to use the same tokens for a long time (they never expired). I will continue using the new tokens I just created, to confirm they don't expire, as they should.

mimijojo commented 3 years ago

Thanks, I'll redo the procedure and hope it won't expire anymore...

gboudreau commented 3 years ago

I myself created a new project to test those updated instructions, and updated my OAuth tokens with that project. I know my last project allowed me to use the same tokens for a long time (they never expired). I will continue using the new tokens I just created, to confirm they don't expire, as they should.

I can confirm the authorization seems to have expired after 7 days. The app I gave access to using the gmvault OAuth process was simply gone from https://myaccount.google.com/permissions It was as-if I never authorized gmvault to access my account... I would guess this is a new limitation added by Google, for non-production (in testing) apps.

mimijojo commented 3 years ago

My token was also revoked after +/- 7 days. I was about to try recreating the token one more time following your instructions even more accurately. According to what you just wrote, this won't help unfortunately. Anyone another hint or workaround to make it persistent ?

yesrod commented 3 years ago

In the Google developer console, under the OAuth consent screen section (https://console.developers.google.com/apis/credentials/consent) my GMVault project had somehow been switch from "Testing" status to "Production" status. Once I switched it back to "Testing" status and added my email to the "Test users" list GMVault started working for me again.

gboudreau commented 3 years ago

A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of "Testing" is issued a refresh token expiring in 7 days.

Ref: https://developers.google.com/identity/protocols/oauth2#expiration

So sadly, until someone can have Google verify their OAuth Consent screen, refresh tokens obtained using gmvault will always expire after 7 days, unless you were lucky enough to already have a "In production" app in your Google developer console before they added this limit, or you are using Google Workspace (and want to backup only domain users).

When the refresh token expires, you just need to run gmvault check --renew-oauth2-tok your_email_address@gmail.com to get a new one. I guess one could do that weekly.

mimijojo commented 3 years ago

Refreshing with "gmvault check --renew-oauth2-tok your_email_address@gmail.com" works indeed ! Only solution is to schedule this manual process once a week... Rather tricky, as my goal was to run this gmvault backup transparently in Docker on my NAS. :-(

maovieira commented 3 years ago

Thank you for the guide. I follow it and successfully backup my emails.

DavidBerdik commented 3 years ago

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense. If you create a new one, if takes a few seconds to created. You will need to manually select it after its been created.
  • Go to https://console.developers.google.com/apis ; click + ENABLE APIS AND SERVICES at the top
  • Search for Gmail API; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials/consent
  • User Type: choose External (`Internal is for Google Workspace accounts; it would limit usage to only accounts within your Workspace)
  • Click Create
  • App name: enter anything you like
  • User support email: choose anything available
  • Developer contact information is also required at the bottom; enter the same email adress
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD OR REMOVE SCOPES
  • Select the checkbox on the Gmail API : https://mail.google.com/ line; click UPDATE
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD USERS
  • Enter all the email addresses you'll want to backup using gmvault (if you want to add more later, come back here and add them)
  • Click ADD
  • Click SAVE AND CONTINUE at the bottom
  • Click Credentials in the left menu
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Desktop app; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create
  • You will get a client ID and secret. Those two values needs to replace the existing ones in $HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and gmvault_client_secret=...
  • If you are running gmvault 1.9.1, make sure conf_version=1.9 in that same conf file, and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change conf_version if you are running 1.9.2-beta-1 or higher.)
  • Finally, obtain a new OAuth token using the following command: gmvault check --renew-oauth2-tok your_email_address@gmail.com When navigating to the URL that gmvault provides, you will see Google hasn’t verified this app. Click Continue, then Allow twice.

Repeat this last step for all other Gmail accounts you are backing up using Gmvault on this particular computer, or the last 3 steps on other computers. (No need to create multiple clientIDs & secrets to backup multiple accounts or for multiple installs.)

This should be pinned so it's easier for newcomers to find. This information is extremely important, but requires some digging right now in order to be found.

DavidBerdik commented 3 years ago

@matthewhelmke It looks like Got-Your-Back also requires you to create a test project. Does it have this issue as well now?

matthewhelmke commented 3 years ago

@DavidBerdik Got-Your-Back is still working for me. I haven't touched my script that calls it in ages, but it runs without error regularly and spot checks of the backup are successful.

DavidBerdik commented 3 years ago

@matthewhelmke Awesome! Thanks! I experimented with moving to Got-Your-Back over the weekend, but I prefer GMvault over it. Fortunately, I've found a way to work around the renewal revocation issue: if you submit the app for verification but don't fill out the necessary forms to actually be formally verified, your project gets put in a state in which it is only available to users that are specified as test users, but since you are also not in test mode anymore, the 7-day expiration no longer happens.

dlmv123 commented 3 years ago

@matthewhelmke Awesome! Thanks! I experimented with moving to Got-Your-Back over the weekend, but I prefer GMvault over it. Fortunately, I've found a way to work around the renewal revocation issue: if you submit the app for verification but don't fill out the necessary forms to actually be formally verified, your project gets put in a state in which it is only available to users that are specified as test users, but since you are also not in test mode anymore, the 7-day expiration no longer happens.

@DavidBerdik How to send for verification ? In the OAuth Consent screen does not have submit for verification button or similar ? Is it to publish app ? Sorry, novice here.

dlmv123 commented 3 years ago

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense. If you create a new one, if takes a few seconds to created. You will need to manually select it after its been created.
  • Go to https://console.developers.google.com/apis ; click + ENABLE APIS AND SERVICES at the top
  • Search for Gmail API; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials/consent
  • User Type: choose External (`Internal is for Google Workspace accounts; it would limit usage to only accounts within your Workspace)
  • Click Create
  • App name: enter anything you like
  • User support email: choose anything available
  • Developer contact information is also required at the bottom; enter the same email adress
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD OR REMOVE SCOPES
  • Select the checkbox on the Gmail API : https://mail.google.com/ line; click UPDATE
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD USERS
  • Enter all the email addresses you'll want to backup using gmvault (if you want to add more later, come back here and add them)
  • Click ADD
  • Click SAVE AND CONTINUE at the bottom
  • Click Credentials in the left menu
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Desktop app; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create
  • You will get a client ID and secret. Those two values needs to replace the existing ones in $HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and gmvault_client_secret=...
  • If you are running gmvault 1.9.1, make sure conf_version=1.9 in that same conf file, and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change conf_version if you are running 1.9.2-beta-1 or higher.)
  • Finally, obtain a new OAuth token using the following command: gmvault check --renew-oauth2-tok your_email_address@gmail.com When navigating to the URL that gmvault provides, you will see Google hasn’t verified this app. Click Continue, then Allow twice.

Repeat this last step for all other Gmail accounts you are backing up using Gmvault on this particular computer, or the last 3 steps on other computers. (No need to create multiple clientIDs & secrets to backup multiple accounts or for multiple installs.)

I have followed the steps but after I paste the verification code, I have the error return as below : Can someone please help ?

Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token. Error: HTTP Error 401: Unauthorized.

=== Exception traceback === Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gmv/gmv_cmd.py", line 743, in run credential = CredentialHelper.get_credential(args) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 235, in get_credential credential = cls.get_oauth2_credential(args['email'], renew) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 387, in get_oauth2_credential access_token, refresh_token, validity, type = cls._get_oauth2_tokens(email, use_webbrowser = True) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 342, in _get_oauth2_tokens response = cls._get_authorization_tokens(verification_code) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 306, in _get_authorization_tokens raise err HTTPError: HTTP Error 401: Unauthorized

=== End of Exception traceback ===

DavidBerdik commented 3 years ago

@dlmv123 I'm not a computer at the moment so I can't check exactly what it is called, but it's somewhere in the Google Console that you use to set up the application. I believe it's under the OAuth credentials page.

Pseudomax commented 2 years ago

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense. If you create a new one, if takes a few seconds to created. You will need to manually select it after its been created.
  • Go to https://console.developers.google.com/apis ; click + ENABLE APIS AND SERVICES at the top
  • Search for Gmail API; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials/consent
  • User Type: choose External (`Internal is for Google Workspace accounts; it would limit usage to only accounts within your Workspace)
  • Click Create
  • App name: enter anything you like
  • User support email: choose anything available
  • Developer contact information is also required at the bottom; enter the same email adress
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD OR REMOVE SCOPES
  • Select the checkbox on the Gmail API : https://mail.google.com/ line; click UPDATE
  • Click SAVE AND CONTINUE at the bottom
  • Click ADD USERS
  • Enter all the email addresses you'll want to backup using gmvault (if you want to add more later, come back here and add them)
  • Click ADD
  • Click SAVE AND CONTINUE at the bottom
  • Click Credentials in the left menu
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Desktop app; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create
  • You will get a client ID and secret. Those two values needs to replace the existing ones in $HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and gmvault_client_secret=...
  • If you are running gmvault 1.9.1, make sure conf_version=1.9 in that same conf file, and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change conf_version if you are running 1.9.2-beta-1 or higher.)
  • Finally, obtain a new OAuth token using the following command: gmvault check --renew-oauth2-tok your_email_address@gmail.com When navigating to the URL that gmvault provides, you will see Google hasn’t verified this app. Click Continue, then Allow twice.

Repeat this last step for all other Gmail accounts you are backing up using Gmvault on this particular computer, or the last 3 steps on other computers. (No need to create multiple clientIDs & secrets to backup multiple accounts or for multiple installs.)

I have followed the steps but after I paste the verification code, I have the error return as below : Can someone please help ?

Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token. Error: HTTP Error 401: Unauthorized.

=== Exception traceback === Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gmv/gmv_cmd.py", line 743, in run credential = CredentialHelper.get_credential(args) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 235, in get_credential credential = cls.get_oauth2_credential(args['email'], renew) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 387, in get_oauth2_credential access_token, refresh_token, validity, type = cls._get_oauth2_tokens(email, use_webbrowser = True) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 342, in _get_oauth2_tokens response = cls._get_authorization_tokens(verification_code) File "/usr/lib/python2.7/site-packages/gmv/credential_utils.py", line 306, in _get_authorization_tokens raise err HTTPError: HTTP Error 401: Unauthorized

=== End of Exception traceback ===

(this is a little late... but for anyone else)

I have just successfully completed the process of connecting GMvault... Note the Google verification code is added into the GMvault terminal You must also check that this is created in the correct Google account (I have 2 and failed the first time as I was logged into the wrong account)

MysticCobra commented 2 years ago

I was excited to find this information, and was following the instructions using a Windows install of GMvault...until I tried my first sync, and got a Google error that reads:

Access blocked: Gmvault’s request is invalid

You can’t sign in because Gmvault sent an invalid request. You can try again later, or contact the developer about this issue. Learn more about this error If you are a developer of Gmvault, see error details. Error 400: invalid_request

Following the error details link gets you here: https://developers.google.com/identity/protocols/oauth2/resources/oob-migration

I'm not a dev and may be misunderstanding, but it looks to me like GMVault is now broken unless there's a code update to use a new API...but also now requires a web server to be running on the machine running GMVault for the authorization to complete. Is that correct??

DavidBerdik commented 2 years ago

@MysticCobra Google deprecated the out-of-band OAuth flow on October 3.

Here are the instructions you need to follow to work around it: https://github.com/gaubert/gmvault/issues/361#issuecomment-1207824886

nicolaskern commented 1 year ago

I have been in a loop of failure, still getting an error after entering this command:

gmvault check --renew-oauth2-tok ****@gmail.com

I get this error after having entered the token:

Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.
Error: HTTP Error 400: Bad Request.

Anybody got this error? Thanks a lot!