search

mholt / timeliner

All your digital life on a single timeline, stored locally -- DEPRECATED, SEE TIMELINIZE (link below)
https://timelinize.com
GNU Affero General Public License v3.0
3.57k stars 116 forks source link

invalid grant from Google Photos #56

Open gdudek opened 4 years ago

gdudek commented 4 years ago

On trying to use timeliner with Google Photos I am constantly getting a Bad Request error as follows:

 Get https://photoslibrary.googleapis.com/v1/albums?pageSize=50&pageToken=: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_grant",
  "error_description": "Bad Request"
} <<< - retrying... (attempt 5/10)

Any ideas?

mholt commented 4 years ago

I'm not sure, never seen that before!

Would be happy to have help if anyone could look into it.

metahnetoutage commented 2 years ago

I will take a look!

metahnetoutage commented 2 years ago

Good News, I was able to provide access by merely added dimabeofek@gmail.com for OAuth consent screen Test users. Granted access.

metahnetoutage commented 2 years ago

Studying how get-all works. Maybe it is related to this issue. This will fix sync issue.

metahnetoutage commented 2 years ago

FYI, if you check my fork, I added the ability to specify the path for the destination folder, so you are always in the know.

metahnetoutage commented 2 years ago

If something of mine is not running as the original this just means I got more work to do so that I behave better than original. Fingers crossed! Trying to make Timeliner beautiful!

metahnetoutage commented 2 years ago

So, several things. Do not fork or pull my repo. Here is what I am up to:1) Fix sync issue on Silicon Mac if possible 2) Add custom repository setup, with a check if it exists. 3) Flatten data so that it is not in folders yet instead split purely by year folders.

metahnetoutage commented 2 years ago

You will notice that I commented out get-latest...why? Because I am testing it. Do not fork yet. I am testing get-all yet must use faster network connection. I did enable Google Picker API as well as Google Photos library. I will see if the error is gone. I also reset the secret after 2 days. Now I have timeliner2->timeliner which I will duplicate and make timeliner3(experimental) testing.

The code checking existence of file which I used can be found here: https://gist.github.com/mattes/d13e273314c3b3ade33f

metahnetoutage commented 2 years ago

On trying to use timeliner with Google Photos I am constantly getting a Bad Request error as follows:

 Get https://photoslibrary.googleapis.com/v1/albums?pageSize=50&pageToken=: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_grant",
  "error_description": "Bad Request"
} <<< - retrying... (attempt 5/10)

Any ideas?

What command did you type? Can you please tell me, I do have a google project called Timeliner I made specifically to debug this app. If you can provide the command you typed I can actually work on it. If I have just nailed getting all photos(I think I have) I can find a faster way to do so, in parallel hence timeliner3. Fix this bug(if it exists).

mholt commented 2 years ago

Hey @metahnetoutage , sorry for the late reply. I see you've been quite active lately. Thanks for learning about this and trying various things, documenting your progress. I haven't had time to look into this recently but it's on my list. Good luck!

metahnetoutage commented 2 years ago

Hi Matt,

I ran into a bug, looking like this:

This is a result of successful authentication of Google Photos, and making a folder at a custom location just like the code says and applying get-all option.

I did get some photos maybe 30-40, including videos until I got this:

2021/11/21 13:07:51 [ERROR] google_photos: https://lh3.googleusercontent.com/lr/AFBm1_b0YMmkRLwBmeBX71o2PuPnZtoc6n0GehgR4VhzvLEytpY3CUilLHHT9h7asa3y903TQoxmUW1RIJonSAUMQrcqpA7C9SxluydhXJ7V6XpTzsP3BCcNz4q-WZAr0TlFNtR1tPjw6cYDmGWgr9t-K-kiTKa0LODu-Ku7Tuy20BqXeGuIWaVvA57t86sjjumXi0dzBl5VxsN0qau95N6debmXOIG4rMjrb8tFt9rCU_Ff_L0HZmQ463Yw6XVoHym2rExR9lTY9k-Hqq2_HGGdpb5RqGKK84kyoml3KeH04_gdQs4qF08VZ-gAacr1ppkd8PDdvDghsbBmXiMRyA-TlQS4eL2-WHuzYNKt0NlIYXgzsVaCqoekoGjlbsKiqjZ5fZ4rDlOTZDC6uCa3zONe_iaqhJIn6UGJDAN30JlWah_t8jLVCQ8A_TLnrDSyv-yty59ZHtjYOMnb5WD5ajMDSzqhFy5C6JRxm8ucCssynZ7DzxCN_xPAK1zOP2VqGzT67OoAwVdpaHJw9pKzw4W6TZun-rY6fiPplGFWXnDsUu46tfKPpD_vVshe9I4AqguoCZEQ4WJ8qvo8VFtWUKHCNUYLIrVczEOrqFg0PQMU1SMR1xvnRMIQNVokK-u2y980NKAAKHNxbaW8KeJ-DErO1BTjcVJOcsdL6e0QHeuPCQIoa3-7mDbx1gSzUUAmMAOmPXiK-DtYx-y00jZj2AKcK7gJ9GWfaeRGkUZzTKvJPfg4sU4J55HECFBieYx7r8iC8-n0Mx2PLrxo5peAAhIVNAAQKXF6KUzNuZOTj0UiiuJlI1C8f6kQQYASY8DkdD2P-FgOiLE=d: getting media contents: Get " https://lh3.googleusercontent.com/lr/AFBm1_b0YMmkRLwBmeBX71o2PuPnZtoc6n0GehgR4VhzvLEytpY3CUilLHHT9h7asa3y903TQoxmUW1RIJonSAUMQrcqpA7C9SxluydhXJ7V6XpTzsP3BCcNz4q-WZAr0TlFNtR1tPjw6cYDmGWgr9t-K-kiTKa0LODu-Ku7Tuy20BqXeGuIWaVvA57t86sjjumXi0dzBl5VxsN0qau95N6debmXOIG4rMjrb8tFt9rCU_Ff_L0HZmQ463Yw6XVoHym2rExR9lTY9k-Hqq2_HGGdpb5RqGKK84kyoml3KeH04_gdQs4qF08VZ-gAacr1ppkd8PDdvDghsbBmXiMRyA-TlQS4eL2-WHuzYNKt0NlIYXgzsVaCqoekoGjlbsKiqjZ5fZ4rDlOTZDC6uCa3zONe_iaqhJIn6UGJDAN30JlWah_t8jLVCQ8A_TLnrDSyv-yty59ZHtjYOMnb5WD5ajMDSzqhFy5C6JRxm8ucCssynZ7DzxCN_xPAK1zOP2VqGzT67OoAwVdpaHJw9pKzw4W6TZun-rY6fiPplGFWXnDsUu46tfKPpD_vVshe9I4AqguoCZEQ4WJ8qvo8VFtWUKHCNUYLIrVczEOrqFg0PQMU1SMR1xvnRMIQNVokK-u2y980NKAAKHNxbaW8KeJ-DErO1BTjcVJOcsdL6e0QHeuPCQIoa3-7mDbx1gSzUUAmMAOmPXiK-DtYx-y00jZj2AKcK7gJ9GWfaeRGkUZzTKvJPfg4sU4J55HECFBieYx7r8iC8-n0Mx2PLrxo5peAAhIVNAAQKXF6KUzNuZOTj0UiiuJlI1C8f6kQQYASY8DkdD2P-FgOiLE=d": read tcp 192.168.68.105:51880->142.250.189.161:443: read: operation timed out - retrying... (attempt 1/5)

2021/11/21 13:07:51 [ERROR] google_photos: https://lh3.googleusercontent.com/lr/AFBm1_ZJIUjFWH6pUmaVraXHHXs1rd1QoVwuq0Qo_Rpp1wq6Dx_jZSGkxAdqIsODhJJ-ggKNxjUqr6AGWz0iNSKGHVXCe_G9a4XJk7rIqEkc5385NqB_KP0frGuBg4fIl13yDdJMNc9CvbfGrSr4k6WDgnMQQ2zpulH5iJgs1CeB0W5aKFx91NRB7wNQOJhmOATQQouZMsA8AHhAgoY8NrJZiaamwEm2BXJQfVRFtsOEizZ2YgFzQZu8-7Bx3Vv8dQwF8qkzkOXQMfI4oirbnoOnHvI_kbBTQTzI1AZFRxqQpOkf2v1-99EnHFfnuvflpvpssNpVP5BbMNYFTrIdzKuyvv-mk58HQu2uju7_pZ99PAFufdhDiOsjEGipYo_XEz8aGrnLhrOi0W2gaPMWZouiwqw3GVXEbkhQN2dQopUbDnz-T_CiCfK_LXCrBIRsmri4TjC2mQ4xOWQEQf54Ps2kUdxNzUOpw0c9gCcf_bEIxGsgpzBpHOQJDy83WL6fVIzMZbMzXOim19JPZFkmzXVffphkpALR4b7gH1Xvw7wPQlehJQgOqI7gQ3lQn2L1O_3-9nJ2e5kT4BQckERWS8cTek0dSWaM2WXm0d0emQ-442aO4bLBztSDgamSWRfS4DbY-J4nIlCNiUpxa04DbmuUfzmCTzAuvJHXrpsE7GDRKrNROVIeDw-J7-SP_SVlnBB1ed-g11cDy9bPyUflTtvsQxgq5BWeBrciemE_H3gJLvFuExjnAwkqZMeauA6vxICdkNGGyt6s2SuJ1CwHYNs1YWNJSvbS_IwIZVY5GRapa3rIYwdvtBXJmXq0QGyWsgbtd-Rw4Eo=d: getting media contents: Get " https://lh3.googleusercontent.com/lr/AFBm1_ZJIUjFWH6pUmaVraXHHXs1rd1QoVwuq0Qo_Rpp1wq6Dx_jZSGkxAdqIsODhJJ-ggKNxjUqr6AGWz0iNSKGHVXCe_G9a4XJk7rIqEkc5385NqB_KP0frGuBg4fIl13yDdJMNc9CvbfGrSr4k6WDgnMQQ2zpulH5iJgs1CeB0W5aKFx91NRB7wNQOJhmOATQQouZMsA8AHhAgoY8NrJZiaamwEm2BXJQfVRFtsOEizZ2YgFzQZu8-7Bx3Vv8dQwF8qkzkOXQMfI4oirbnoOnHvI_kbBTQTzI1AZFRxqQpOkf2v1-99EnHFfnuvflpvpssNpVP5BbMNYFTrIdzKuyvv-mk58HQu2uju7_pZ99PAFufdhDiOsjEGipYo_XEz8aGrnLhrOi0W2gaPMWZouiwqw3GVXEbkhQN2dQopUbDnz-T_CiCfK_LXCrBIRsmri4TjC2mQ4xOWQEQf54Ps2kUdxNzUOpw0c9gCcf_bEIxGsgpzBpHOQJDy83WL6fVIzMZbMzXOim19JPZFkmzXVffphkpALR4b7gH1Xvw7wPQlehJQgOqI7gQ3lQn2L1O_3-9nJ2e5kT4BQckERWS8cTek0dSWaM2WXm0d0emQ-442aO4bLBztSDgamSWRfS4DbY-J4nIlCNiUpxa04DbmuUfzmCTzAuvJHXrpsE7GDRKrNROVIeDw-J7-SP_SVlnBB1ed-g11cDy9bPyUflTtvsQxgq5BWeBrciemE_H3gJLvFuExjnAwkqZMeauA6vxICdkNGGyt6s2SuJ1CwHYNs1YWNJSvbS_IwIZVY5GRapa3rIYwdvtBXJmXq0QGyWsgbtd-Rw4Eo=d": read tcp 192.168.68.105:51880->142.250.189.161:443: read: operation timed out - retrying... (attempt 1/5)

I will look into it today, any idea is that is a fetch attempt after session timeout (session is shorter than the code expects, or a result of losing connection, or cause of a bad link as in, hear me out:

{ photo_video_data, link, amendedLink} and say your program looks only at link, so Apple looks at say amendedLink or link+amendedLink, so because of such difference I get an error.

Just a thought,

Dima

On Mon, Nov 22, 2021 at 11:01 AM Matt Holt @.***> wrote:

Hey @metahnetoutage https://github.com/metahnetoutage , sorry for the late reply. I see you've been quite active lately. Thanks for learning about this and trying various things, documenting your progress. I haven't had time to look into this recently but it's on my list. Good luck!

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975828115, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZVUIV4CPT7ODQXDWMTUNKHQTANCNFSM4LWGWB7Q .

mholt commented 2 years ago

I see those sometimes too, it's just the network (or their servers) being slow, that's why it retries.

metahnetoutage commented 2 years ago

So to avoid it, is it enough to do a pause on our end or and let them catch up? What topic should I study?

Dima

On Mon, Nov 22, 2021 at 11:30 AM Matt Holt @.***> wrote:

I see those sometimes too, it's just the network (or their servers) being slow, that's why it retries.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975851387, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZTVMS6ZYSJPAMIZM2LUNKK7DANCNFSM4LWGWB7Q .

metahnetoutage commented 2 years ago

Is there a way currently to do fixed amount of photo or video?

Or you must amend code for that?

Dima

On Mon, Nov 22, 2021 at 11:33 AM Dima Dimov @.***> wrote:

So to avoid it, is it enough to do a pause on our end or and let them catch up? What topic should I study?

Dima

On Mon, Nov 22, 2021 at 11:30 AM Matt Holt @.***> wrote:

I see those sometimes too, it's just the network (or their servers) being slow, that's why it retries.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975851387, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZTVMS6ZYSJPAMIZM2LUNKK7DANCNFSM4LWGWB7Q .

mholt commented 2 years ago

Can probably adjust timeouts in the dialer for the HTTP client. Look at the net/http and net packages for docs.

metahnetoutage commented 2 years ago

If you check the errors that I changed you will see I aimed to make them client and developer friendly. Hopefully they can be merged no problem and if there is any error amendment that should be changed, or clarified let me know please.

Dima On Mon, Nov 22, 2021 at 11:35 AM Matt Holt @.***> wrote:

Can probably adjust timeouts in the dialer for the HTTP client. Look at the net/http and net packages for docs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975854425, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZSJ3U3W3Z7ONWGIZ2DUNKLPHANCNFSM4LWGWB7Q .

metahnetoutage commented 2 years ago

My first priority is fixing as many of 26 bugs as possible.

What are your thoughts about using mongoDB instead of sql3? Also, I am considering integrating this for iCloud for timeliner, thoughts: https://github.com/metahnetoutage/icloud

I forked it from another developer who forked it from author, yet developer did fixes author did not. We should be able to use this, depends on their license of course.

metahnetoutage commented 2 years ago

I have turned timeliner2 into timeliner2 and did a copy. I will do a mongoDB example and show you. If you like it, we can proceed with it.

metahnetoutage commented 2 years ago

Interesting find, so I plugged 142.250.189.161 into Safari and I get Google.

What the error below means in my opinion is, we are very generous with retries, instead of being wise about it. Meaning, Google say has the right to say, excuse me, my session for this moment is done. timeliner:EHEM....GIVE ME THE VIDS AND PHOTOS....WAAAH! ;)

This is the logic I would follow, if we are approaching google timeout window, request a new time slice, Get the length of data, and keep aggregating it, as we approach time limit, request a new time slice, and continue this until data is fully transferred.

The same process should be for any data.

192.168.68.105:51880->142.250.189.161:443: read: operation timed out

Thoughts?

metahnetoutage commented 2 years ago

Looking at NewRateLimitedRoundTripper from ratelimit.go, will test in timeliner3, if I nail it, I will apply to my repo!

metahnetoutage commented 2 years ago

The reason I do not contribute to your project just yet is because I am working on a Trie, and studying Go(finishing Udemy course), and studying MySQL. As soon as I get Go certificate & MySQL certificate I will fix any bugs that I see and help fix the rest.

Dima

On Mon, Nov 22, 2021 at 1:24 PM metahnetoutage @.***> wrote:

Looking at NewRateLimitedRoundTripper from ratelimit.go, will test in timeliner3, if I nail it, I will apply to my repo!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975932059, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZSYDE6375EGRF7LPQTUNKYKTANCNFSM4LWGWB7Q .

metahnetoutage commented 2 years ago

Hi Matt,

Is there a way that you can scrape the current timeliner project and put what you are currently working on?(let us say there be a free and a paid version and GitHub will be free.

I would love to work and fix if anything what you are working on. I will happily delete timeliner project. It does use go, as is likely your current project.

I stopped doing Go while I am wrapping my own projects. Yet I will get back to Go in around 5 months. Before that I will finish MongoDB course on Udemy and MySQL as well.

I can make timeliner work…I just do not want my efforts to be in vain since I rather work on something that you will keep and everyone will use.

Let me know what you think.

I have recently done a Google Drive folder listener and will soon download specific data to a database and display it. This way someone can change content in a folder and that content be displayed on the site.

I purposely took a break to learn databases and finish ago before I fix or edit and figure out what is broken.

Sincerely,

Dima

On Tue, Jan 18, 2022 at 12:25 PM metahnetoutage @.***> wrote:

The reason I do not contribute to your project just yet is because I am working on a Trie, and studying Go(finishing Udemy course), and studying MySQL. As soon as I get Go certificate & MySQL certificate I will fix any bugs that I see and help fix the rest.

Dima

On Mon, Nov 22, 2021 at 1:24 PM metahnetoutage @.***> wrote:

Looking at NewRateLimitedRoundTripper from ratelimit.go, will test in timeliner3, if I nail it, I will apply to my repo!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-975932059, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALE26ZSYDE6375EGRF7LPQTUNKYKTANCNFSM4LWGWB7Q

.

— Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-1015805329, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZQAVUUXZ4D3GTC55L3UWXECHANCNFSM4LWGWB7Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

mholt commented 2 years ago

@metahnetoutage Thanks for the update. Sorry I haven't been very responsive.

The code for Google Photos authentication hasn't changed in Timelinize at this point (and yes, it is written in Go). So a contribution here can still be applied to the future result!

Can I send you an email about collaborating on it? What's your email address?

metahnetoutage commented 2 years ago

@.***

On Mon, Jul 25, 2022 at 9:35 AM Matt Holt @.***> wrote:

@metahnetoutage https://github.com/metahnetoutage Thanks for the update. Sorry I haven't been very responsive.

The code for Google Photos authentication hasn't changed in Timelinize at this point (and yes, it is written in Go). So a contribution here can still be applied to the future result!

Can I send you an email about collaborating on it? What's your email address?

— Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-1194327587, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZUJPVKJOKQUWA7TQPLVV266TANCNFSM4LWGWB7Q . You are receiving this because you were mentioned.Message ID: @.***>

mholt commented 2 years ago

@metahnetoutage Can you please comment on GitHub. When you reply from an email it redacts your address.

metahnetoutage commented 2 years ago

@.***

On Mon, Jul 25, 2022 at 1:57 PM Matt Holt @.***> wrote:

@metahnetoutage https://github.com/metahnetoutage Can you please comment on GitHub. When you reply from an email it redacts your address.

— Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-1194620323, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZR7LIC3CRVNK2EQFATVV35TRANCNFSM4LWGWB7Q . You are receiving this because you were mentioned.Message ID: @.***>

metahnetoutage commented 2 years ago

I left my email in a reply to an issue.

On Mon, Jul 25, 2022 at 2:13 PM metahnetoutage @.***> wrote:

@.***

On Mon, Jul 25, 2022 at 1:57 PM Matt Holt @.***> wrote:

@metahnetoutage https://github.com/metahnetoutage Can you please comment on GitHub. When you reply from an email it redacts your address.

— Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-1194620323, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALE26ZR7LIC3CRVNK2EQFATVV35TRANCNFSM4LWGWB7Q

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/mholt/timeliner/issues/56#issuecomment-1194636490, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZS4WX2P23FCN4QUTJLVV37OFANCNFSM4LWGWB7Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>