search

Magnitus- / gogcli

Client to Interact With the API of GOG.com
MIT License
40 stars 0 forks source link

SkipUrls broken in v0.22.0 #23

Closed xKevin04 closed 1 year ago

xKevin04 commented 1 year ago

I tried to generate a manifest and got a few 404 URLs again, so I excluded them via regex pattern in the SkipUrls array of the manifest-generation-progress.json file - as explained here.

But something breaking must have changed between v0.21.0 and v0.22.0, because the new version seemingly ignores this option, while it works fine with the old one, just as before. I tried manifest generate-resume on both versions with the exact same progress file containing the following filter option:

      "SkipUrls": [
        "^/downloads/episode_2_blackwell_unbound/en2installer0$",
        "^/downloads/episode_3_blackwell_convergence/en2installer0$",
        "^/downloads/heart_of_the_woods/en1patch1$",
        "^/downloads/the_blackwell_epiphany/en2installer0$"
      ],

v0.21.0 completed without errors, but v0.22.0 terminated pretty quickly after reaching the point where it actually processes games. It printed the following error before exiting:

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/episode_2_blackwell_unbound/en2installer0) -> body download handle retrieval error: did not expect status code of 404

That error should not have happened if SkipUrls were honored. I did several tries and it stopped on episode 3 a few times as well, even though the exact URL was also configured to be skipped. The progress file was initially created by v0.22.0, so a format change can't be the reason (I updated gogcli before doing my quarterly backup).

These are all the broken URLs from my library right now, so things got better in that regard. Just one of the old patch links left and three Mac installers that will probably get resolved.

On a side note, manifest generation is blazingly fast in comparison to before, because there wasn't a single file that needed the workaround method to get its checksums. Seems like their CDN problems and having to switch to their own temporary solution was good for one or two things at least (not sure if it had anything to do with it, but sounds plausible to me). Just ~1.5 hours for a complete manifest on v0.21.0 with no errors beyond these four broken URLs, almost ideal conditions for my library.

Magnitus- commented 1 year ago

fyi, I created the following shell script to troubleshoot your issue in a temporary subdirectory of the project:

(cd ../..; go build .)
cp ../../gogcli .
cp ../../cookie .
./gogcli manifest generate -r 1 --title=Blackwell

I can confirm that running the script gave me this:

./bugsetup.sh 
[manifest writer] Generating/Updating manifest for 5 games
[manifest writer] Got all info on game with id 1207662883. 4 games left to process
 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/episode_2_blackwell_unbound/en2installer0) -> body download handle retrieval error: did not expect status code of 404

And I can confirm that after adding the skip url in the progress file like so:

      "SkipUrls": ["^/downloads/episode_2_blackwell_unbound/en2installer0$"],

And running the resume command:

./gogcli manifest generate-resume -r 1
[manifest writer] Generating/Updating manifest for 4 games
 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/episode_2_blackwell_unbound/en2installer0) -> body download handle retrieval error: did not expect status code of 404

The url indeed wasn't skipped and should have been so there is a regression there. I'll look into why that is.

Separately, I'll file a ticket with gog.com for them to fix their broken installer link.

And yes, the experience with their CDN has improved significantly. Very grateful for that as well. I was getting discouraged at how long it was taking.

Magnitus- commented 1 year ago

Alright, you should find that your issue is resolved with release 1.22.1.

btw, all the following installer files don't download currently:

      "^/downloads/episode_2_blackwell_unbound/en2installer0$",
      "^/downloads/episode_3_blackwell_convergence/en2installer0$",
      "^/downloads/the_blackwell_epiphany/en2installer0$"

I filed a support request with gog for that.

xKevin04 commented 1 year ago

Thank you! I can confirm it's fixed in v0.22.1 :)