search

gaubert / gmvault

gmail backup software
http://www.gmvault.org
GNU Affero General Public License v3.0
3.58k stars 288 forks source link

Google will revoke Gmvault access March 31, 2019 #335

Closed matthewhelmke closed 5 years ago

matthewhelmke commented 5 years ago

FYI, I just received this email. I've been using Gmvault for a few years assuming it was abandoned, since there have been no commits since 2016, but thought I would create this issue anyway, just in case...

Hello,

Although you don’t need to take any action, we wanted to let you know that the following third-party apps will no longer be able to access some data in your Google Account, including your Gmail content. This change will go into effect starting March 31, 2019.

Gmvault

We are making this change as part of ongoing efforts to make sure your data is protected and private. These apps haven’t yet complied with our updated data privacy requirements announced on October 8, 2018.

You can always view, manage and remove apps you’ve given access to your account by visiting your Google Account.

Thanks, The Google Accounts team

tofurky commented 5 years ago

nice. remove our ability to archive our emails to help protect it 😂

joshstrange commented 5 years ago

I’m hoping this can worked around by replacing an api token somewhere in the source and/or requiring users to register their own app with google. It’s not ideal but it’s easy enough and free for our level of usage I would guess.

I looked a little in the code but I’m on mobile so I’ll have to look later when I’m on my laptop.

tofurky commented 5 years ago

see https://github.com/gaubert/gmvault/blob/ee915c2799927a30e95963a9c406b415f4825cce/src/gmv/credential_utils.py#L38-L58

joshstrange commented 5 years ago

@tofurky Wow, I was in the file and even scrolled over but didn’t scroll far enough to see the “default values”. I even dug in gmvaul_utils a little after that before I gave up on my phone lol.

That gives me hope that we can work around this, potentially, if I’m scanning that code right, with just a config/defaults file?

johanatan commented 5 years ago

I am interested in whatever solution is ultimately reached here.

fortran77 commented 5 years ago

Would somebody please find a procedure for users to get their own token, and post the steps required? Then each person doesn't have to figure it out separately.

joshstrange commented 5 years ago

I started generating new credentials and this was interesting. I'm guessing this the option that gmvault uses:

Option 3: Manual copy/paste

Important: The custom URI scheme and loopback IP address options now provide more reliable, secure, and user-friendly ways to obtain user authorization. This option may be discontinued in the future and should only be used if the choices above are not viable.

I'm still reading but:

Option 2: Loopback IP address (macOS, Linux, Windows desktop)

To receive the authorization code using this URL, your application must be listening on the local web server. That is possible on many, but not all, platforms. However, if your platform supports it, this is the recommended mechanism for obtaining the authorization code.

When your app receives the authorization response, for best usability it should respond by displaying an HTML page that instructs the user to close the browser and return to your app.

This looks like a realistic option for us if option 3 is ever removed. I think it will be but I also think we have more time. Right now we can focus on getting to to work past March 31st.

gboudreau commented 5 years ago

The client_id and secret are in $HOME/.gmvault/gmvault_defaults.conf Swapping them there should do the trick without needing to modify the code.

joshstrange commented 5 years ago

Ok this should be as easy as replacing 2 lines in you config (~/.gmvault/gmvault_defaults.conf)

[GoogleOauth2]
scope=https://mail.google.com/
# The URL root for accessing Google Accounts.
google_accounts_base_url=https://accounts.google.com
# Hardcoded dummy redirect URI for non-web apps.
redirect_uri=urn:ietf:wg:oauth:2.0:oob
#identifier and secret in app mode for gmvault
gmvault_client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com
gmvault_client_secret=IVkl_pglv5cXzugpmnRNqtT7

Those last 2 lines are what we are going to need to change. I generated 2 new ones but it will only allow for 100 "Sensitive logins" unless my "app" is approved. I don't really want to be the one responsible for this going forward and maybe it's best instead of publishing new keys we write a guide on how to generate your own? It literally takes less than 5 minutes.

joshstrange commented 5 years ago

It literally takes less than 5 minutes.

Note: I already had an account setup, I just had to create a new "Project" so I don't know how long it would take for someone who had never used the google API console or if they will have to setup billing info (My account does have billing on file).

tofurky commented 5 years ago

falling back to generating an app password and using that is an option. i think that should bypass any of the oauth stuff. example:

$ gmvault sync --store-passwd user@gmail.com
<enter app password>
^C
$ gmvault sync -p user@gmail.com
...

see https://support.google.com/accounts/answer/185833?hl=en for details on app passwords. not ideal, though.

joshstrange commented 5 years ago

@tofurky Yeah, I saw that mentioned in a different issue. That might be the easier long-term solution.

I wish github made it easy to see if any of the 200+ forks on this saw any real development. I'd switch to a fork of this if I knew there was at least some housekeeping going on.

brechmos commented 5 years ago

It sounds like there is some direction on fixes for this.

When there is an acceptable solution could someone post in the Wiki and then post back here of the "accepted" solution?

(Glad there are lots thinking and working on this!)

[Edit, Hah, for some reason I was able to post in the future...] image

gboudreau commented 5 years ago

Procedure to get yourself a new client ID & secret:

Repeat this last step for all other Gmail accounts you are backing up using Gmvault on this particular computer, or the last 3 steps on other computers. (No need to create multiple clientIDs & secrets to backup multiple accounts or for multiple installs.)

Someone pointed out that:

It seems that there is a way to circumvent the 7 days expiration for the refresh token. You have to publish the application and then skip sending any verification details. Just clicking the publish button and creating a new refresh token is enough (sometimes you need to recreate again after 7 days in order for it to work).

Pectojin commented 5 years ago

FYI, it seems google is pretty sticky with handing out the API access, so it may take some time to get this working.

Screenshot 2019-03-22 at 00 23 00 Screenshot 2019-03-22 at 00 32 13
brechmos commented 5 years ago

There is a "Submit for verification" beside the "save" button. Not sure if that speeds up the process at all...

DavoOZ commented 5 years ago

Questions: 1) Does anyone have any idea of how often one's access credentials would need to be updated (using the manual method described above)? https://github.com/gaubert/gmvault/issues/335#issuecomment-475437988

2) My reading of the new Google API conditions suggests that the actual app may need to be accredited for access, as well as individual users - are I correct in this?

chbug commented 5 years ago

I’m hoping this can worked around by replacing an api token somewhere in the source and/or requiring users to register their own app with google. It’s not ideal but it’s easy enough and free for our level of usage I would guess.

I looked a little in the code but I’m on mobile so I’ll have to look later when I’m on my laptop.

I'm confused: why is there a need to work around it instead of getting the current key reviewed once and for all? gmvault doesn't move user data to a cloud storage or anything where the data of multiple users gets centralized, so it should be straightforward, no?

daneroo commented 5 years ago

I successfully created App credentials with @gboudreau 's great instructions above. Thanks for taking the time to write that up, (and being so thorough!)

dan86603 commented 5 years ago

Daneroo - did you get confirmation that you won't have access issues or will you have to wait until after March 31st?

Thanks

daneroo commented 5 years ago

Not sure @dan20047 ,

fortran77 commented 5 years ago

@joshstrange and others interested, here is a list of forks with a last commit date in 2017 and later. Now you can search for ones with the most current development. Anything useful should preferably be merged back here if possible.

2019:

https://github.com/Anmol-Singh-Jaggi/gmvault https://github.com/mbarczak/gmvault

2018:

https://github.com/fossabot/gmvault https://github.com/hengy2017/gmvault

2017:

https://github.com/andriusadamonis/gmvault https://github.com/asez73/gmvault https://github.com/csirmaz/gmvault https://github.com/eliask/gmvault

dygordon commented 5 years ago

Answers re: the consent screen/verification here: https://support.google.com/cloud/answer/7454865

In brief:

1) you can only mark an app "internal" if you have G Suite and you are creating it within your organization, in which case it is internal by default.

2) 100 sensitive logins is defined as 100 new users, so this limit shouldn't matter anyway if it's just you. I think. Not sure if it will see each use of the OAuth credentials by gmvault as a new user or not. But some rclone forums setting up G Drive scopes suggest no.

EDIT:

Well, now that I actually try the login, I get the same "Sign in with Google temporarily disabled" screen some have reported despite not being anywhere close to the 100 new user cap.

Looking at the URL and then the file, looks like gmvault is reverting the changes to $HOME/.gmvault/gmvault_defaults.conf and putting its own app keys back in. I guess this is the known issue at #245 and #273 and some of you may be running the beta version as I am on 1.9.1.

@gboudreau @daneroo you may want to check if your changes to the conf file actually did stick or you perhaps just reauthed using the standard gmvault keys (client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com).

seanlane commented 5 years ago

In regards to the issue with $HOME/.gmvault/gmvault_defaults.conf being overwritten, one quick patch is to change the line conf_version=1.9.1 to conf_version=1.9, per issue #273. I wasn't able to find any documentation on the differences between config files of the two versions, but some initial testing hasn't found problems. Feel free to correct me if this would actually cause any issues though.

gboudreau commented 5 years ago

@dygordon Yes, my changes did stick. But while I am using version 1.9.1 of gmvault, my .conf shows 1.9 (because it was created when I was running 1.9 I would guess), so like @seanlane pointed out, that seems to prevent the overwriting problem from happening. I'll add that to the guide I posted above.

yesrod commented 5 years ago

I overlooked a critical step at first; I needed to move or delete the old oauth2 token and repeat the sign-in process.

I'm using the docker image from https://hub.docker.com/r/aubertg/gmvault-docker so for me the process was: 1) Follow the instructions in gboudreau's comment to generate and install new credentials 2) cd to the directory mounted as /data in the container 3) move email@gmail.com.oauth2 to email@gmail.com.oauth2.old 4) attach to the existing container with docker exec -it gmvault-docker /bin/bash 5) run su -c 'gmvault sync -d /data ${GMVAULT_EMAIL_ADDRESS}' gmvault and follow the instructions

jaskerr commented 5 years ago

Followed @gboudreau instructions for new client ID and secret.

Google's Oauth2 endpoint: https://accounts.google.com/o/oauth2/token returned Error: HTTP Error 401: Unauthorized.

Had to remove the old Oauth2 token and generate a new one (similar to @yesrod).

Might want to update instructions to include the generation of new tokens.

benreic commented 5 years ago

Just used @gboudreau instructions and worked perfectly. They've been updated with the feedback from the successive comments so the instructions are comprehensive.

tidhub commented 5 years ago

Thank you @gboudreau, could you please clarify what the name should be:

  • Fill name, click Add scope <-- is name = Gmvault, or can I give it a name?
  • Select the checkbox on the https://mail.google.com/ line; click Add
  • Click Save
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Other; give it a name <-- is this name the same as the one above?
gboudreau commented 5 years ago

@tidhub Both names can be anything you'd like.

tidhub commented 5 years ago

Great thank you @gboudreau

jezzabart commented 5 years ago

Many thanks to @gboudreau for your very clear instructions. I have no idea what they do, but they worked perfectly.

dalin- commented 5 years ago

I ran @gboudreau 's instructions: https://github.com/gaubert/gmvault/issues/335#issuecomment-475437988

But when I ran gmvault I got the error:

Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.

Error: HTTP Error 400: Bad Request.

=== Exception traceback ===
Traceback (most recent call last):
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.gmv_cmd", line 743, in run
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 235, in get_credential
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 411, in get_oauth2_credential
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 267, in _get_oauth2_acc_tok_from_ref_tok
HTTPError: HTTP Error 400: Bad Request

The solution was related to @yesrod's instructions https://github.com/gaubert/gmvault/issues/335#issuecomment-476654974 but a bit different since I'm not using Docker:

  1. cd ~/.gmvault/
  2. mv <my-email>.oath2 <my-email>.oath2.bak
  3. Re-run Gmvault, follow the instructions about authenticating.
medmunds commented 5 years ago

Note that if you need to back up both GSuite accounts and actual \@gmail.com addresses, you must create your gmvault project using your \@gmail.com Google account, not a GSuite account. (In the first step of @gboudreau's instructions, check the Google account switcher at top right of https://console.developers.google.com/.)

It seems that, for unpublished client ids:

BTW, after you've updated your gmdefaults.conf, an easy way to obtain new auth tokens is gmvault check --renew-oauth2-tok $YOUR_EMAIL_ADDRESS

limes007 commented 5 years ago

The procedure of @gboudreau has worked, together with the "renew" command of @medmunds. Thank you very much. I wonder if or how long my Google project will be free of charge (and maybe how long this will work at all). As far as I understand, its a free trail only. Does anyone know for sure?

cornasdf commented 5 years ago

Hi all, Thanks for the write ups, i am back online with this.

FYI, when at the 'Create OAuth client ID' step, if you choose web application, you will end up with the error below when trying to do the web page redirect/auth flow. I was successful when I chose 'Other' as the application type at https://console.developers.google.com/apis/credentials/oauthclient

Error: redirect_uri_mismatch

The redirect URI in the request, urn:ietf:wg:oauth:2.0:oob, can only be used by a Client ID for native application. It is not allowed for the WEB client type. You can create a Client ID for native application at https://console.developers.google.com/apis/credentials/oauthclient
dsl101 commented 5 years ago

I'm using 1.9.2-beta-1 on windows, and in the end I had to make gmvault_defaults.conf unwriteable (Properties→Security→Edit, Select User, click Write→Deny) to stop gmvault putting it's own keys back in. Once I'd done that, I was able to authenticate.

lexelby commented 5 years ago

This looks like a workable replacement for gmvault, including backing up based on a gmail search: https://github.com/jay0lee/got-your-back/

They seem to have run into the same authentication issue, and they've partially automated and fully streamlined the process of setting up your google project to get auth working.

mhsmith commented 5 years ago

Like @dsl101 above, I'm also using 1.9.2-beta-1 on Windows. However, making gmvault_defaults.conf unwriteable did not work for me. It did stop the file from being overwritten, but gmvault still considered it to be out of date and ignored it, so my client ID and secret weren't used.

Instead, I simply left conf_version at its previous value of 1.9.2-beta-1. Aside from that, I was able to follow the instructions without change.

dsl101 commented 5 years ago

Just to clarify, I didn't change conf_version—also left at 1.9.2-beta-1.

gitasong commented 5 years ago

I'm going through @gboudreau's instructions. When I try to save the config file with my new client ID and secret, I get Unable to save file: Permission denied '/.gmvault'. I've never modified a Unix executable before and have no idea how to properly do so. Can anyone advise?

gitasong commented 5 years ago

Found it. For Mac users, the config file is located at /Users/yourname/.gmvault/gmvault_defaults.conf. It'll give you a No Storage DB in /Users/nicolefreed/gmvault-db. Create it. message if you try to run the OAuth update command gmvault check --renew-oauth2-tok your_email_address@gmail.com, but just ignore it and start backing up e-mails with ./gmvault sync myuser@gmail.com. Gmvault should work fine at that point.

mfonville commented 5 years ago

Would anyone please fork the project and put out a new (maybe renamed) release that incorporates setting your own gmvault_client_id and gmvault_client_secret during the setup process? and also to directly tackle #273 :-)

samadore commented 5 years ago

Thank @gboudreau for much needed step-by-step, but when I run last command to get new token it returns "command not found". Clearly, I'm doing something wrong. Any idea what?

luckylittle commented 5 years ago

Thanks @gboudreau, your instructions https://github.com/gaubert/gmvault/issues/335#issuecomment-475437988 worked for me.

I am not sure how these restrictions will affect us/me. I will never be able to submit it for verification - because it asks for Application's authorized DNS domain that you own, application homepage link and application privacy policy link - both must be hosted on an that authorized domain. Github.com is obviously not possible to use.

atcroft commented 5 years ago

If one follows @gboudreau 's notes, and uses Gmvault to backup more than one Gmail account, does one have to go through the process on both accounts?

If so, would there not be separate client_id and client_secret values? Would one need to be able to specify separate configs that contained the appropriate value for the account being used? (And how could they best do so?)

If not, which part of the notes would need to be executed on the second (or following) accounts?

BechirZalila commented 5 years ago

Thank you very much. This worked perfectly

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense
  • Go to https://console.developers.google.com/apis ; click + ENABLED APIS AND SERVICES at the top
  • Search for Gmail; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials; choose OAuth Consent Screen tab
  • Fill name (enter anything), click Add scope
  • Select the checkbox on the https://mail.google.com/ line; click Add
  • Click Save
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Other; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create
  • You will get a client ID and secret. Those two values needs to replace the existing ones in $HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and gmvault_client_secret=...
  • If you are running gmvault 1.9.1, make sure conf_version=1.9 in that same conf file, and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change conf_version if you are running 1.9.2-beta-1 or higher.)
  • Finally, obtain a new OAuth token using the following command: gmvault check --renew-oauth2-tok your_email_address@gmail.com
gboudreau commented 5 years ago

@atcroft You create the client ID and secret only once. You then use them with as much as 100 different accounts by putting them in the gmvault_defaults.conf file of the computers that you use to backup Gmail accounts. If you only use one computer, only the last step needs to be repeated for multiple accounts.

mhasteley commented 5 years ago

Has anyone had the instructions above, by @gboudreau work since August? I set up the project as indicated in the (impressively exhaustive) instructions (setting the app as 'internal' so as to avoid the google review requirement) but I'm still presented with the app verification screen when logging a user account in for gmvault oauth.

I am using 1.9.1 with the read-only work around from the other thread (changing it to 1.9 threw a different error).

matthewhelmke commented 5 years ago

I originally opened this issue. There have been many good suggestions and workarounds mentioned in the thread. Meanwhile, I switched to this: https://github.com/jay0lee/got-your-back/. It is working well for me.

With that, I'm going to close this thread. Best wishes to all and thanks to those who commented with helpful ideas!