perkeep / gphotos-cdp

This program uses the Chrome DevTools Protocol to drive a Chrome session that downloads your photos stored in Google Photos.
Apache License 2.0
658 stars 36 forks source link

Downloading in "..." took too long to start (238GB and 30408 files, over ~2TB) #41

Closed prigal closed 11 months ago

prigal commented 11 months ago

Hello,

I'm starting to have issues while downloading my library. I can't figure if it's because I have too many folders in my library and the program is hanging or because of something else.

I'm not a Go developer but I have an env up and running and I'm able to build and add some logs to the program.

This is a trace :

2023/12/10 10:51:19 Session Dir: /tmp/gphotos-cdp
2023/12/10 10:51:20 pre-navigate
2023/12/10 10:51:22 post-navigate
2023/12/10 10:51:23 Page loaded, most recent item in the feed is: AF1QipNDfV7M14cFI66EfUHLzzMEqskDttntw7SQZpRq
2023/12/10 10:51:24 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:24 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:28 Marking https://photos.google.com/photo/AF1QipOaVU56JChJFU-3RafrUU9q5cA_nEMYqIPpW3yQ as done
2023/12/10 10:51:29 Event: {Type:keyDown Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:51:29 Event: {Type:keyUp Modifiers:Shift Timestamp:<nil> Text: UnmodifiedText: KeyIdentifier: Code:KeyD Key:D WindowsVirtualKeyCode:68 NativeVirtualKeyCode:68 AutoRepeat:false IsKeypad:false IsSystemKey:false Location:0 Commands:[]}
2023/12/10 10:56:29 downloading in "/media/prigal/TOSHIBA2T/gphoto" took too long to start

It looks like it's not able to enter this loop

for _, v := range entries {
    if v.IsDir() {
        continue
    }
    if v.Name() == ".lastdone" {
        continue
    }
    if v.Name() == ".lastdone.bak" {
        continue
    }
    fileEntries = append(fileEntries, v)
}

I tried to allocate more time to deadline but with no more luck

deadline := time.Now().Add(time.Minute * 5)

Any idea of what could go wrong ?

NB : I have already downloaded 238GB over ~2TB and I have 30408 folders in the destination folder.

prigal commented 11 months ago

I just checked the location to see what file is blocking, and this is what google photo is showing :

image

Maybe the problem is on their side in the end.

prigal commented 11 months ago

For future readers, some files may be corrupted on google side, and the program will crash trying to download them.

To make it continue, go to your google photo, copy the link of the next image/video in the list, put it in your .lastdone file, the program will resume, download this new file and continue.