search

perkeep / perkeep

Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
https://perkeep.org/
Apache License 2.0
6.5k stars 448 forks source link

importer: add Google Drive importer #896

Open bradfitz opened 7 years ago

bradfitz commented 7 years ago

It seems the Google Photos API (the "Picasaweb API") is broken and doesn't let you fetch all your photos. Details in #874.

But a workaround is to enable Google Drive's setting to mirror Google Photos in your Drive storage. (Go to https://drive.google.com/drive/my-drive then the Gear at the right, Settings > "Create a Google Photos folder" -> [X] Automatically put your Google Photos into a folder in My Drive)

Even if it weren't just for Google Photos, a Google Drive importer would be nice anyway, so let's do it.

/cc @mpl

mpl commented 7 years ago

Yep I've been wanting a drive importer anyway for a while too.

(on phone)

On Dec 30, 2016 02:06, "Brad Fitzpatrick" notifications@github.com wrote:

It seems the Google Photos API (the "Picasaweb API") is broken and doesn't let you fetch all your photos. Details in #874 https://github.com/camlistore/camlistore/issues/874.

But a workaround is to enable Google Drive's setting to mirror Google Photos in your Drive storage. (Go to https://drive.google.com/ drive/my-drive then the Gear at the right, Settings > "Create a Google Photos folder" -> [X] Automatically put your Google Photos into a folder in My Drive)

Even if it weren't just for Google Photos, a Google Drive importer would be nice anyway, so let's do it.

/cc @mpl https://github.com/mpl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/camlistore/camlistore/issues/896, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmNl7vT6bIwV_EHut8Qdf17Q5uN8KzZks5rNFkwgaJpZM4LX8Sp .

tgulacsi commented 7 years ago

The v3 api and the helps seems quite usable (https://developers.google.com/drive/v3/web/quickstart/go, https://godoc.org/google.golang.org/api/drive/v3).

If we use the drive.DrivePhotosReadonlyScope, then no other filtering is needed. list-files.zip

tgulacsi commented 7 years ago

I've started experimenting with it: https://github.com/tgulacsi/gphotos If anyone has any progress, shoot me down!

mpl commented 7 years ago

For the importer to be useful for me (in itself, regardless of the google photos issue), I would need to be able to specify "directories", and/or files, to ignore. Does that seem like a reasonable feature to everyone?

Also, as opposed to our previous importers, some of the items in the drive (except for photos) are getting regularly updated, so how do we handle that? I mean, since we're content-addressable, it would be pretty easy for us to keep all versions of a file, but do we want to do that? I would prefer "overwriting" (i.e. dereferencing, so it eventually gets GCed) past versions of a file. Especially since drive has its own history/revert mechanism anyway. WDYAT?

tgulacsi commented 7 years ago

An alpha quality minimal version is available at https://camlistore-review.googlesource.com/9308 This works just as the Picasa importer, gets only the photos ("photos" space).

What kind of directories would you ignore (under the "photos" space)?

Iff all photos has been successfully imported, the changes are received through the "Changes" service, with the stored token, so receiving the changes are quite possible.

The dereferencing is new to me, so help wanted there...

mpl commented 7 years ago

@tgulacsi thanks, I'll try to have a look soon.

I don't think I'd need/want to ignore anything under the "photos space" as you say. I meant that if we're doing this, we might want to make it a generic gdrive importer anyway (not just for photos), in which case there are indeed files and directories that I'd like to ignore in my drive.

But maybe we only want photos for now if making a generic gdrive importer is much harder. we'll see.

ianlewis commented 6 years ago

Since Google Photos has a proper API now, is it maybe worth using that? https://developers.google.com/photos/

mholt commented 6 years ago

Just be aware that the new API strips a lot of the valuable metadata and exif data out of the photos and videos... If that's important then it's not yet a great solution.

mpl commented 6 years ago

@ianlewis excellent news, thanks. If this new API allows us to to actually get the latest version of the stored photo (as opposed to with the drive API, which is bugged that way), then it's definitely worth it.

mpl commented 6 years ago

@mholt Do you have an example or some documentation/report about that please?

mpl commented 6 years ago

Btw, none of these last comments change the fact that we also eventually want a drive importer for its own merit (not just for photos) anyway, which is kindof what this issue is about now.

For photos themselves, we should probably talk about it on https://github.com/perkeep/perkeep/issues/1144

mholt commented 6 years ago

@mpl Yes, I've been tracking it in https://github.com/mholt/photobak/issues/20.

Documentation: https://developers.google.com/photos/library/reference/rest/v1/mediaItems

Here's the bug report: https://issuetracker.google.com/issues/111228390

They did start including some metadata for photos, but not videos. Here's the bug report for adding metadata for videos: https://issuetracker.google.com/issues/112010631

Still, it's probably better than nothing, unless you're already getting all the metadata using old APIs, in which case, switching to the new API would likely be a net loss.

ivucica commented 5 years ago

Agreed with @mpl, especially given the Photos<->Drive integration is seemingly going poof: https://github.com/perkeep/perkeep/issues/1144#issuecomment-502461294