search

mholt / photobak

Back up your content from Google Photos - DEPRECATED: use Timeliner
https://github.com/mholt/timeliner
309 stars 29 forks source link

Use the new Google Photos API #20

Closed mholt closed 5 years ago

mholt commented 6 years ago

Announced this week: https://developers.google.com/photos/

Getting started guide: https://developers.google.com/photos/library/guides/get-started

michielvandergeest commented 6 years ago

Would this be something that can resolve the 10.000 photo limit?

mholt commented 6 years ago

Theoretically, yes. The only limits I know of are here: https://developers.google.com/photos/library/guides/api-limits-quotas

mholt commented 6 years ago

So, I've been playing with this and this API seems quite nice. WAY nicer than the Picasa Web Albums API. This API would simplify a lot of things (as long as it continues to work as well as I've seen so far).

However, it returns a lot less metadata. For example, it doesn't have location information. It also strips most of the EXIF data. Interestingly, the docs do mention a location field, but currently they say: "Not yet available. Location of the media item."

Hopefully they will add the missing data, because right now it's really great in that the original (and edited) photos can be downloaded, with all their EXIF and metadata intact.

I still hope to go forward on this, because I guess having the photos (and having them reliably) is better than maybe having them with a lot of extra metadata (although I do view location and EXIF as important, I'd REALLY like to have those).

If we go forward with that, I'm considering changing how Photobak organizes the media. Instead of downloading into individual albums and then cross-referencing them in other albums (complicated!), I'm thinking of just downloading them as a single collection into folders as bins of, say, 1000 or 500 images each. We could still have an index of which photos are in which albums stored in the database, but it won't be as readily available on disk.

Still, the point of this program is to make an efficient archive, not a mirror of the Google Photos app.

Any thoughts?

ondrejsevcik commented 6 years ago

I agree, the metadata are almost as important as the photo itself, but it's better to have a reliable backup than nothing. As long as the photos will have correct date, I don't care much about folder structure.

mholt commented 6 years ago

Good news: the missing EXIF metadata is a bug, meaning it's more likely to get fixed: https://issuetracker.google.com/issues/111228390

I think I'll feel comfortable using the new API when it preserves EXIF data in the actual photo. We can always extract that and save it separately if we want to. I will still need to do some checking on photo edits, etc, before committing to that, though.

I'm still leaning towards restructuring the storage, which will simplify a lot of logic.

michielvandergeest commented 6 years ago

Sorry. A bit late to get back to this issue.

The way I see it:

I'm not really used to programming in Go, but if there's an easy task or anything else that I can do to help, would be happy to do so.

I'm getting close to 10.000 photos, and as far as i understand it, if I add older photos to Google Photos (cleaning up old hard disks), they won't get backed up with Photobak, right? So looking forward to a new release based on this new API :)

mholt commented 6 years ago

Organization of photos in folders of 1000 or 5000: I don't see this as a problem. Just wondering how the upgrade would work?

I'm not sure. I'm afraid it might be a start-from-scratch proposition. 😕I just don't know a good way to correlate the photos from the old API with the new one to know what is already downloaded, etc.

I'm not really used to programming in Go, but if there's an easy task or anything else that I can do to help, would be happy to do so.

Thanks! I'll let you know. But for now I'm just waiting until EXIF metadata is restored.

I'm getting close to 10.000 photos, and as far as i understand it, if I add older photos to Google Photos (cleaning up old hard disks), they won't get backed up with Photobak, right? So looking forward to a new release based on this new API :)

Right. Crossing fingers 🤞

michielvandergeest commented 6 years ago

Graceful upgrade path would be nice, but I imagine it being quite a complex endeavour to get right. For me personally it's OK to have to download the photos again, if that facilitates a new release based on the new API.

Hoping they'll fix the EXIF metadata bug soon then :)

mholt commented 6 years ago

Woo hoo! These Google engineers know how to ship: https://issuetracker.google.com/issues/111228390

We have just released a new version of the Google Photos Library API that now supports this feature.

Unfortunately, it appears that it doesn't include location metadata and is only for photos (not videos) so far. Stay tuned...

Johnlon commented 6 years ago

Bug fixed aug 1st?

gidoBOSSftw5731 commented 6 years ago

I like the code here and I want to use it in a personal project, but I want to use the new API, what would be required to do that?

mholt commented 6 years ago

@Johnlon Unfortunately the new API is still missing some information I consider to be pretty essential, like location information. It also doesn't affect videos, which still are missing all EXIF information (last I checked).

@gidoBOSSftw5731 I'm planning to rewrite the Google Photos provider in the googlephotos package. You could certainly do that (there's a lot of details to work out). However, I've elected not to do it quite yet since the new API is missing critical metadata.

mpl commented 6 years ago

@mholt I'm a bit confused by the wording: now when you use the d parameter as they suggested, do you get the whole actual photo as it was originally uploaded (with all its metadata within unaffected), or not? As far as I'm concerned, that's all that matters (for Perkeep), since we can extract all the interesting metadata from the original file.

gidoBOSSftw5731 commented 6 years ago

Last I checked the video one is set to never be fixed, a priority/severity of P3S3

On Sun, Sep 30, 2018, 17:44 Matt Holt notifications@github.com wrote:

@Johnlon https://github.com/Johnlon Unfortunately the new API is still missing some information I consider to be pretty essential, like location information. It also doesn't affect videos, which still are missing all EXIF information (last I checked).

@gidoBOSSftw5731 https://github.com/gidoBOSSftw5731 I'm planning to rewrite the Google Photos provider in the googlephotos package. However, I've elected not to do it quite yet since the new API is missing critical metadata.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mholt/photobak/issues/20#issuecomment-425754555, or mute the thread https://github.com/notifications/unsubscribe-auth/AMpJuZkzqQo1rxOh7ElEeVcQi0-_pdRBks5ugTtYgaJpZM4T7p8L .

mholt commented 5 years ago

@mpl As of the current API, you get the photo as presented in Google Photos, not the original that you uploaded; all EXIF data appears to have been reconstructed with location explicitly stripped out. See https://developers.google.com/photos/library/reference/rest/v1/mediaItems#MediaItem.

Not looking good for preserving the locations of your memories. :(

A little birdie told me that the Google Drive API doesn't strip the location metadata out, but I believe you have to be using Drive space for the storage of your photos in that case, and I've heard about lots of problems with management of a Drive-mapped photo library. :-/

mpl commented 5 years ago

@mholt if you want to access the photos through Drive, (so you can get them either through the Drive API or through any Drive client, the web interface included) you indeed have to tick some option to make it so, but I don't think it changes anything regarding your storage quota if that's what you were worried about. If I look at the properties of one of my photos through the Drive web interface, it shows as occupying 0 bytes of storage.

On the other hand, it is true that it comes with its shares of problems. Many people are reporting sync related issues, such as some of their photos missing, or only appearing months later. And as I've stated before, the main problem for me/us at Perkeep is that through the Drive API you get the photo as it was initially backed up to Google Photos, and none of any subsequent modifications (cropping, rotating, adding filters, etc) are taken into account.

gidoBOSSftw5731 commented 5 years ago

AFAIK, drive is a mirror and therefore any effect made to the drive copy wont change your photos in photos.google.com

mpl commented 5 years ago

@gidoBOSSftw5731 Sure. This is not what I was talking about. See https://productforums.google.com/forum/#!msg/drive/HbNOd1o40CQ/VfIJCncyAAAJ

gidoBOSSftw5731 commented 5 years ago

I see, maybe just make a copy of the photo when you're done cropping/editing it?

mpl commented 5 years ago

Ah yes, thanks. I had some success with various tricks like deleting/re-uploading, but maybe I didn't try just making a copy, I forget.

mholt commented 5 years ago

So, I've got a PoC hodge-podge of code working that uses the new Google Photos API. It's way easier to use and much more reliable than the Picasa Web Albums API.

Except, as discussed in this thread earlier, Google Photos excludes the most critical metadata (other than timestamp): location.

The lack of location is really disappointing. I'm bummed. I just hope they will make it available, but that's unlikely. 😢

The new application I'm developing is inspired by and will probably replace photobak. For years now I've had a grander vision of organizing content from online services, and photobak was just a stepping stone, and backing up photos is just one part of my vision...

I hope I'll have something to push sooner or later. It'll likely be in a new repository.

mholt commented 5 years ago

I've just realized that what I'm building ("new project") is very similar to Perkeep:

And the similarities have bothered me enough that I'm up at 4am writing this to get it off my mind.

After 🤦‍♂️at first, I now think there's still a few significant differences that justify a separate project:

  1. Perkeep is highly focused on its blob storage mechanisms, and remote accessibility, etc., to essentially create your own home-based cloud storage. Really cool tech, but not photobak's / "my new project's" focus. "New project" instead focuses on item and people relationships, and the idea of a single unified timeline.

  2. Perkeep has swappable storage backends, "new project" does not. (Darn. Open to contributions to make that happen though!)

  3. "New project" dumps data files to disk in its original form, organized in folders by date, instead of using a custom blob format that is opaque without special tooling. This makes browsing larger chunks of data like images and videos much easier and more natural, since no special tools/software are required to read the data. You can just open your file explorer and browse away. (Text data or any data that can easily fit in a database table is stored there instead, so it gets indexed. Long-form text data can be stored on disk still, and it's totally flexible depending on how the service is integrated.)

  4. Perkeep appears to be primarily configured via a web UI. (Very cool!) "New project" may be, but is first and foremost designed as a library. (And maybe Perkeep is too, but I'm not clear from the docs...) "New project" has a CLI wrapper and GUIs for it may be implemented for it, of course, but I'm not sure my initial release will have one.

  5. Perkeep's Google Photos importer uses either Picasa Web Albums API (as photobak does currently) or Google Drive -- I don't yet see the new Google Photos API being used. (Maybe because it lacks location data. I dunno.) Here is their importers package: https://github.com/perkeep/perkeep/tree/master/pkg/importer

  6. "New project" provides facilities for making service integrations really easy and efficient: checkpointing, rate limiting, OAuth2, etc. Its OAuth2 facility can even be configured to work through a proxy, for cases where you are hosting your own app that uses "new project" (I'm thinking of integrating this with another project of mine, hence the OAuth2 proxy feature -- but totally optional). Checkpointing allows long-running tasks to be resumed if interrupted (but is optional). Rate limiting is also taken care of for each service integration. The way that service integrations add items to "new project" scales efficiently with the number of items so memory use does not get too high (tasks don't need to be built up and batched; items and groups of items can be processed as they flow in).

Anyway, although some of these differences are surface-level, I'm hoping that it's just different enough that it justifies my work on it, I'm too far done to stop now...

(Also, FYI: I'm planning on developing integrations for Google Photos, Facebook, Twitter, Google Location History, Gmail, and Instagram, to meet my own needs. But of course any other integrations may be contributed. Data sources don't even have to be web apps: they can be anything that can "list items" or be imported from an archive file having a standard layout/format.)

mholt commented 5 years ago

The Picasa Web Albums API is now deprecated and is turned off: https://developers.google.com/picasa-web/docs/3.0/deprecation

I think it's time to sunset this project, too.

Fortunately, Timeliner is here: https://github.com/mholt/timeliner - it fulfills all my needs (despite the lack of location data, which can be obtained other ways) and is the true successor to Photobak.

I will be taking steps to archive this repository over the next little bit.