search

Kalanyr / gogrepoc

Python-based tool for downloading all your GOG.com game and bonus collections to your local computer for full offline enjoyment.
233 stars 36 forks source link

Games that can't be downloaded / have persistent non-fatal warnings though Gogrepoc but can be downloaded though the GOG website #60

Open Kalanyr opened 2 years ago

Kalanyr commented 2 years ago

Please use this thread to report games / items, that have problems downloading through gogrepoc but do download through the website (please include the website link you used to verify this (ie the one with /en0game1 or /18102 at the end, not the specific download URI ) Please separate out reports of items that do not download vs those that have persistent non-fatal errors.

theg721 commented 2 years ago

I'm not 100% sure if this is the same issue as I'm not trying to download anything, just run the update command, but the following games in my library all have XML parsing errors + they can all be downloaded successfully via the website:

Hope this is the link format you were after—if not let me know, happy to provide any further details as necessary.

Full output is here: https://pastebin.com/JDZjczNL (also includes some request failed errors?)

Ygg01 commented 2 years ago

AI war 2 had a fail during update https://gist.github.com/Ygg01/9405cfb46dd214edad82e5422bb29d4b

Jaysyn904 commented 2 years ago

Cyberpunk 2077

gogrepo.log

alzhur commented 2 years ago

No man's sky (maybe error on gog side?):

15:31:52 | (1 / 1) fetching game details for no_mans_sky...
15:32:11 | xml parsing error occurred trying to get md5 data for setup_no_mans_sky_3.84_sentinel_84011_(54108)-1.bin
15:32:11 | The handled exception was:
15:32:11 | 
Traceback (most recent call last):
  File "./gogrepoc.py", line 692, in fetch_file_info
    shelf_etree = xml.etree.ElementTree.fromstring(md5_response.content)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1312, in XML
    return parser.close()
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1665, in close
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: no element found: line 1, column 0
15:32:11 | End exception report.
15:32:27 | xml parsing error occurred trying to get md5 data for setup_no_mans_sky_3.84_sentinel_84011_(54108)-3.bin
15:32:27 | The handled exception was:
15:32:27 | 
Traceback (most recent call last):
  File "./gogrepoc.py", line 692, in fetch_file_info
    shelf_etree = xml.etree.ElementTree.fromstring(md5_response.content)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1312, in XML
    return parser.close()
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1665, in close
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: no element found: line 1, column 0
15:32:27 | End exception report.
pasbec commented 1 year ago

I was able to solve all md5 issues for files that can manually be downloaded from GOG by one small modification in the function append_xml_extension_to_url_path:

diff --git a/gogrepoc.py b/gogrepoc.py
index 98cd8a1..968d92b 100755
--- a/gogrepoc.py
+++ b/gogrepoc.py
@@ -898,7 +898,7 @@ def is_numeric_id(s):

 def append_xml_extension_to_url_path(url):
     parsed = urlparse(url)
-    return urlunparse(parsed._replace(path = parsed.path + ".xml"))
+    return urlunparse(parsed._replace(path = parsed.path + ".xml")).replace('%28','(').replace('%29',')')

 def process_argv(argv):
     p1 = argparse.ArgumentParser(description='%s (%s)' % (__appname__, __url__), add_help=False)

Seems like GOG does not like the encoding of braces when requesting xml files in some cases.

Kalanyr commented 1 year ago

Ahh. Thank you. I'm guessing what it doesn't like is the brackets that are actually part of the file name being replaced.

On Wed, 19 Oct 2022, 02:18 pasbec, @.***> wrote:

I was able to solve all md5 issues for files that can manually be downloaded from GOG by one small modification in the function append_xml_extension_to_url_path:

diff --git a/gogrepoc.py b/gogrepoc.py index 98cd8a1..968d92b 100755--- a/gogrepoc.py+++ b/gogrepoc.py@@ -898,7 +898,7 @@ def is_numeric_id(s):

def append_xml_extension_to_url_path(url): parsed = urlparse(url)- return urlunparse(parsed._replace(path = parsed.path + ".xml"))+ return urlunparse(parsed._replace(path = parsed.path + ".xml")).replace('%28','(').replace('%29',')')

def process_argv(argv): p1 = argparse.ArgumentParser(description='%s (%s)' % (appname, url), add_help=False)

Seems like GOG does not like the encoding of braces when requesting xml files in some cases.

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1282655440, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ334SVCHGWJKQ24B3ENDWD3ET5ANCNFSM5RBM5O2Q . You are receiving this because you authored the thread.Message ID: @.***>

Sargon-Aelther commented 1 year ago

Ahh. Thank you. I'm guessing what it doesn't like is the brackets that are actually part of the file name being replaced.

Sorry if it's a dumb question, but do you plan to implement pasbec's discovery?

Kalanyr commented 1 year ago

Yes. I'm just migrating between computers and phones at the moment so it'll probably be a week or two before I get my programming environment setup again.

On Thu, 20 Oct 2022, 02:37 Sargon-Aelther, @.***> wrote:

Ahh. Thank you. I'm guessing what it doesn't like is the brackets that are actually part of the file name being replaced.

Sorry if it's a dumb question, but do you plan to implement pasbec's discovery?

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1284288273, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ333URXB23HBUBFUQU23WEAPVPANCNFSM5RBM5O2Q . You are receiving this because you authored the thread.Message ID: @.***>

Wilstorm commented 1 year ago

I don't know if this error belongs in this thread or if it's enough information, just let me know if it's incorrect as I am new to the script. I have 733 games to download but it fails on game

(11 / 733) fetching game details for alien_isolation...

I tried multiple times and the error is always the same and it always the same game. The error

request failed: HTTPSonnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s....

The first 10 games take a few seconds and everything seems to be working fine, the 11 just hangs for a bit, then it's just screens of error after error so I quit the script.

On a side note, I have 746 games in Galaxy but when I start downloading it only shows 733 to download.

Also, what a neat script and thank you for writing it! :)

Kalanyr commented 1 year ago

Thanks.

The 13 missing games are problem ones you've hidden in your library. Those don't get updated / downloaded by default, but there is a flag you can set to force them to be if you want.

On Thu, 20 Oct 2022, 08:19 Wilstorm, @.***> wrote:

I don't know if this error belongs in this thread or if it's enough information, just let me know if it's incorrect as I am new to the script. I have 733 games to download but it fails on game (11 / 733) fetching game details for alien_isolation... I tried multiple times and the error is always the same and it always the same game. The error request failed: HTTPSonnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s.... The first 10 games take a few seconds and everything seems to be working fine, the 11 just hangs for a bit, then it's just screens of error after error so I quit the script.

On a side note, I have 746 games in Galaxy but when I start downloading it only shows 733 to download.

Also, what a neat script and thank you for writing it! :)

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1284631758, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ332MTOCKBSK2GOTC5WLWEBXYNANCNFSM5RBM5O2Q . You are receiving this because you authored the thread.Message ID: @.***>

pasbec commented 1 year ago

Hi @Kalanyr, may I suggest to add some --verbose switch (true/false would do) to the main argument parser and use this to enabled/disable the detailed expection logging like this:

if args.verbose:
    warn("The handled exception was:")
    log_exception('')
    warn("End exception report.")

I repeatedly see that users are terminating the script or think there is some unhandeled error due to the detailed excpetion log. The optional verbose option should be fairly simple to implement and would still allow you or developers to debug the code or tell other users to use it if they need help.

What do you think?

Kalanyr commented 1 year ago

Yeah, I should move that to debug level, rather than it's current level.

On Fri, Oct 21, 2022 at 12:16 AM pasbec @.***> wrote:

Hi @Kalanyr https://github.com/Kalanyr, may I suggest to add some --verbose switch (true/false ould do) to the main argument parser and use this to enabled/disable all the places like this:

if args.verbose: warn("The handled exception was:") log_exception('') warn("End exception report.")

I repeatedly see that users are terminating the script or think there is some unhandeled error due to the detailed excpetion log. The verbose would still allow you to debug the code or tell other users to run with --verbose if they need help.

What do you think?

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1285626962, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ33322AMFAJURKDOSU4DWEFH5NANCNFSM5RBM5O2Q . You are receiving this because you were mentioned.Message ID: @.***>

pasbec commented 1 year ago

I don't know if this error belongs in this thread or if it's enough information, just let me know if it's incorrect as I am new to the script. I have 733 games to download but it fails on game

(11 / 733) fetching game details for alien_isolation...

I tried multiple times and the error is always the same and it always the same game. The error

request failed: HTTPSonnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s....

The first 10 games take a few seconds and everything seems to be working fine, the 11 just hangs for a bit, then it's just screens of error after error so I quit the script.

On a side note, I have 746 games in Galaxy but when I start downloading it only shows 733 to download.

Also, what a neat script and thank you for writing it! :)

There seem to be still many games on GOG, where md5/xml information is missing. If you always end up with timeouts for one specific game, you'll probably hitting such an issue. @Kalanyr has already created a seperate issue to collect a list of affected games but there's probably not much that can be done from our side. The only solution that I can see is to contact GOG, let them know and hope they get it fixed.

Wilstorm commented 1 year ago

@Kalanyr @pasbec - Thank you for the reply and help. Just a quick question. If I do contact GOG what would I say? I am guessing they don't support any script issues. If that's a completely wrong I apologize for the assumption. I would gladly reach out to them and mention any issues that might help but I'm not quite sure what to point out as the issue. Gogrepoc download issues for the specific game or missing MD5/XML information? Thanks again guys for the quick replies and help.

Kalanyr commented 1 year ago

I was able to solve all md5 issues for files that can manually be downloaded from GOG by one small modification in the function append_xml_extension_to_url_path:

diff --git a/gogrepoc.py b/gogrepoc.py
index 98cd8a1..968d92b 100755
--- a/gogrepoc.py
+++ b/gogrepoc.py
@@ -898,7 +898,7 @@ def is_numeric_id(s):

 def append_xml_extension_to_url_path(url):
     parsed = urlparse(url)
-    return urlunparse(parsed._replace(path = parsed.path + ".xml"))
+    return urlunparse(parsed._replace(path = parsed.path + ".xml")).replace('%28','(').replace('%29',')')

 def process_argv(argv):
     p1 = argparse.ArgumentParser(description='%s (%s)' % (__appname__, __url__), add_help=False)

Seems like GOG does not like the encoding of braces when requesting xml files in some cases.

I have implemented this locally and am currently testing it. So far it seems like it helps a lot (no connection errors so far from 56 games + fewer content decoding errors ), but I've had one content decoding error so need to drill down and see if that's a different problem.

That's still a substantial enough improvement to justify integrating it, so thank you.

Sargon-Aelther commented 1 year ago

I assume this is still being tested and is not part of the recent update?

Kalanyr commented 1 year ago

Think Prison Architect is the only one of these I own and my check revealed it was only the 32bit windows installers so it's a pretty low priority for me ATM.

If these are more like Imperator Rome (where the servers are desynced, so a single test doesn't actually reveal the problem) then let me know.

Also please separate out Fails from Warnings.

I'll walk you through filing a support request with GOG for random fails, ie how to phrase it, what info to include etc.

On Thu, 17 Nov 2022, 11:55 Sargon-Aelther, @.***> wrote:

I assume this is still being tested and is not part of the recent update?

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1317944496, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ335KAVGVOABQ6EMFEGLWIWGAZANCNFSM5RBM5O2Q . You are receiving this because you were mentioned.Message ID: @.***>

pasbec commented 1 year ago

Thank you very much for keeping this nice tool running and updated @Kalanyr!

BileDemon commented 1 year ago

This is the complete list of games in my library that gogrepoc fails to fetch/download:

a_plague_tale_innocence
absolute_drift
alien_isolation
alien_isolation_-_the_trigger
batman_arkham_origins
battlevoid_harbinger
blade_runner
brigador_up_armored_edition
call_of_juarez
chicago_1930_the_prohibition
crosscode_ninja_skin
deadlock_planetary_conquest
dream_pinball_3d
dying_light_the_following_enhanced_edition
fallout_new_vegas
heroes_of_might_and_magic_v_-_tribes_of_the_east
imperium_galactica
ironclad_tactics
mafia_definitive_edition
mafia_iii_definitive_edition
metro_exodus
nikopol_secrets_of_the_immortals
openttd_12.2
parkan_the_imperial_chronicles
post_mortem_1.0
project_warlock
raiden
resonance
return_to_mysterious_island
riven_-_the_sequel_to_myst
sacred_2_gold
silent_storm_gold
space_empires_4_deluxe
starbound_1.4.2
stronghold_hd
sudden_strike_resource_war
sudden_strike_3
technobabylon_2_5a
terminator_resistance
the_last_blade_2_cm
the_mystery_of_the_druids
witcher3_de_goty
patch_the_witcher_3_wild_hunt_-_game_of_the_year_edition
thronebreaker_1.1
time_gate_knight_s_chase_de_gog
tomb_raider_1_german
trine_3_the_artifacts_of_power
unavowed_
the_undergames_v2.1
warhammer_skulls_2022_digital_goodie_pack
wasteland_2_directors_cut_digital_deluxe_edition_upgrade
wolfenstein_the_new_order
wolfenstein_the_old_blood
x3_reunion
zafehouse_diaries

All these games fail with either one of these error messages:

xml parsing error occurred trying to get md5 data for ...
unexpected connection error fetching md5 data for ...
unexpected content decoding error fetching md5 data for ...
request failed: 403 Client Error: Forbidden for url: ...
request failed: 404 Client Error: Not Found for url: ...

The errors are quite numerous. I didn't want to download all 55 missing games through the GOG website, so I can't verify if this works yet. I guess it should work that way. At least I hope that the XML Parsing/Decoding problems can be resolved in a future version.

Kalanyr commented 1 year ago

These shouldn't actually stop downloads of anything real for the record.

403 should be files that don't exist and aren't expected too (ie preorders and Kickstarter rewards that were never made available on GOG)

The md5 errors are "just" the error checking files and aren't required to download the file just to check that the file isn't subtly broken.

On Sun, 22 Jan 2023, 01:43 BileDemon, @.***> wrote:

This is the complete list of games in my library that gogrepoc fails to fetch/download:

a_plague_tale_innocence absolute_drift alien_isolation alienisolation-_the_trigger batman_arkham_origins battlevoid_harbinger blade_runner brigador_up_armored_edition call_of_juarez chicago_1930_the_prohibition crosscode_ninja_skin deadlock_planetary_conquest dream_pinball_3d dying_light_the_following_enhanced_edition fallout_new_vegas heroes_of_might_and_magicv-_tribes_of_the_east imperium_galactica ironclad_tactics mafia_definitive_edition mafia_iii_definitive_edition metro_exodus nikopol_secrets_of_the_immortals openttd_12.2 parkan_the_imperial_chronicles post_mortem_1.0 project_warlock raiden resonance return_to_mysteriousisland riven-_the_sequel_to_myst sacred_2_gold silent_storm_gold space_empires_4_deluxe starbound_1.4.2 stronghold_hd sudden_strike_resource_war sudden_strike_3 technobabylon_2_5a terminator_resistance the_last_blade_2_cm the_mystery_of_the_druids witcher3_de_goty patch_the_witcher_3_wildhunt-_game_of_the_year_edition thronebreaker_1.1 time_gate_knight_s_chase_de_gog tomb_raider_1_german trine_3_the_artifacts_ofpower unavowed the_undergames_v2.1 warhammer_skulls_2022_digital_goodie_pack wasteland_2_directors_cut_digital_deluxe_edition_upgrade wolfenstein_the_new_order wolfenstein_the_old_blood x3_reunion zafehouse_diaries

All these games fail with either one of these error messages:

xml parsing error occurred trying to get md5 data for ... unexpected connection error fetching md5 data for ... unexpected content decoding error fetching md5 data for ... request failed: 403 Client Error: Forbidden for url: ... request failed: 404 Client Error: Not Found for url: ...

The errors are quite numerous. I didn't want to download all 55 missing games through the GOG website, so I can't verify if this works yet. I guess it should work that way. At least I hope that the XML Parsing/Decoding problems can be resolved in a future version.

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1399275550, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ3356ZKKYYNDBE5QPDQDWTP73JANCNFSM5RBM5O2Q . You are receiving this because you were mentioned.Message ID: @.***>

BileDemon commented 1 year ago

Oh ... you are absolutely right. I totally misunderstood what these messages meant. I was under the impression every error represented a single file that could not be downloaded.

Thank you for the clarification.

idkicarus commented 1 year ago

The script gives me a 404 error when trying to fetch the base game and patch files for Imperator: Rome. The extras and DLC are added to the manifest and downloaded without issue.

16:41:19 | (1 / 1) fetching game details for imperator_rome...
16:41:25 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1installer0.  will not retry.
16:41:25 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1installer0
16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1installer1.  will not retry.
16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1installer1
16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch0.  will not retry.
16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch0
16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch1.  will not retry.
16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch1
16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch2.  will not retry.
16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch2
16:41:27 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch3.  will not retry.
16:41:27 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch3

I can download these files from GOG's website without any problems.

The current base game files are:

The current patch files are:

Kalanyr commented 1 year ago

This is a longstanding GOG issue where their download servers are out of sync, you'll need to update -ids imperator_rome (and any other flags you use) until it no longer generates errors, then you can just move the orphaned files back to avoid having to fight with the downloads.

I've reported this 4 times now and it's been fixed on a couple of them but it keeps coming back.

On Fri, 7 July 2023, 06:53 idkicarus, @.***> wrote:

The script gives me a 404 error when trying to fetch the base game and patch files for Imperator: Rome. The extras and DLC are added to the manifest and downloaded without issue.

16:41:19 | (1 / 1) fetching game details for imperator_rome... 16:41:25 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1installer0. will not retry. 16:41:25 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1installer0 16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1installer1. will not retry. 16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1installer1 16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch0. will not retry. 16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch0 16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch1. will not retry. 16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch1 16:41:26 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch2. will not retry. 16:41:26 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch2 16:41:27 | request failed: 404 Client Error: Not Found for url: https://www.gog.com/downloads/imperator_rome/en1patch3. will not retry. 16:41:27 | failed to fetch https://www.gog.com/downloads/imperator_rome/en1patch3

I can download these files from GOG's website without any problems.

The current base game files are:

  • setup_imperator_rome_2.0.3rc2(45615).exe
  • setup_imperator_rome_2.0.3rc2(45615)-1.bin

The current patch files are:

  • patch_imperator_rome1.5.1(40458)to_1.5.2(40614).exe
  • patch_imperator_rome1.5.2(40614)to_1.5.3(40956).exe
  • patch_imperator_rome_2.0.1RC2(44994)to_2.02_RC1(45377).exe
  • patch_imperator_rome_2.02RC1(45377)to_2.0.3_RC2(45615).exe

— Reply to this email directly, view it on GitHub https://github.com/Kalanyr/gogrepoc/issues/60#issuecomment-1624295660, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKZ332RDB54HM2SB7ODCBDXO4QS7ANCNFSM5RBM5O2Q . You are receiving this because you were mentioned.Message ID: @.***>

idkicarus commented 1 year ago

Sure enough, that worked. I tried again this morning and retrieved the files. Thanks for the quick response!