completely rewrite GameCache

[don-de-marco]

@Autistic-unicorn may I once more call you in as tester because of your massive steam library? just let it run and let me know if it breaks and how the performance feels (i didn't touch the achievement import yet, so no improvements there unfortunately)

[Autistic-Unicorn]

Unfortunately I can't test it right now. I'm gonna get into it as soon as I'm gonna have time tomorrow :)

[engiefox]

I tried and it crashed while trying to import games from the old cache plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log GalaxyClient.log

[don-de-marco]

@Autistic-Unicorn no worries, I'm off to bed already and won't be able to respond within a timely manner anyway^^

@engiefox can you post the log lines containing the exception (preferably including the backtrace)? I'm on my phone atm and cannot open the log files there

[engiefox]

2023-06-18 16:38:29,536 - steam_network.websocket_client - ERROR - Failed to establish authenticated WebSocket connection AttributeError("'SteamLicense' object has no attribute 'packagie_id'") 2023-06-18 16:38:29,538 - steam_network.websocket_client - ERROR - Traceback (most recent call last): File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\websocket_client.py", line 111, in run await run_task File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol_client.py", line 113, in run await self._protobuf_client.run() File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 178, in run await self._process_packet(packet) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 518, in _process_packet await self._process_message(emsg, header, packet[8 + header_len:]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 525, in _process_message await self._process_multi(body) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 565, in _process_multi await self._process_packet(data[offset + size_bytes:offset + size_bytes + size]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 518, in _process_packet await self._process_message(emsg, header, packet[8 + header_len:]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 535, in _process_message await self._process_license_list(body) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 672, in _process_license_list await self.license_import_handler(licensed_apps) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol_client.py", line 368, in _license_import_handler await self._protobuf_client.get_packages_info(packages_to_import) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 440, in get_packages_info await self._send(EMsg.ClientPICSProductInfoRequest, message) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 482, in _send body = bytes(message) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 629, in __bytes__ wraps=meta.wraps or "", File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 360, in _serialize_single value = _preprocess_single(proto_type, wraps, value) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 346, in _preprocess_single return bytes(value) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 438, in <lambda> message.packages = map(lambda x: CMsgClientPICSProductInfoRequestPackageInfo(x.packagie_id, x.access_token), steam_licenses) AttributeError: 'SteamLicense' object has no attribute 'packagie_id'

[engiefox]

I believe that's what you want, the steam log cuts off at the bottom, let me know if you need something else. There's also these lines in the client log: 2023-06-18 16:38:45.200 [Warning][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Failed to retrieve product auth info for '1207659142' while checking for updates. 2023-06-18 16:38:45.505 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Setting product event RecentlyChecked for product '1207662473' for 00min. 2023-06-18 16:38:45.811 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Retrieved valid product authorisation for '1207662473' from the database. 2023-06-18 16:38:45.811 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Fetching product configuration for '1207662473'. 2023-06-18 16:38:46.114 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Serializing product configuration (cloud saves configuration is present, overlay configuration is present) for '1207662473'. 2023-06-18 16:38:46.323 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Setting product event RecentlyChecked for product '1436454557' for 00min. 2023-06-18 16:38:46.828 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: There is not any authorisation values for '1436454557' in the database. 2023-06-18 16:38:46.828 [Warning][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Failed to retrieve product auth info for '1436454557' while checking for updates. 2023-06-18 16:38:47.031 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Setting product event RecentlyChecked for product '1584866499' for 00min. 2023-06-18 16:38:47.340 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Retrieved valid product authorisation for '1584866499' from the database. 2023-06-18 16:38:47.340 [Information][SubSystem installer thread (4)] [TID 20888][galaxy_client]: Fetching product configuration for '1584866499'. 2023-06-18 16:38:47.456 [Error][ (0)] [TID 6140][galaxy_client]: Plugin - pluginID/steam_ca27391f-2675-49b1-92c0-896d43afa4f8 is not responding. 2023-06-18 16:38:47.456 [Information][ (0)] [TID 6140][galaxy_client]: Request to stop plugin pluginID/steam_ca27391f-2675-49b1-92c0-896d43afa4f8 2023-06-18 16:38:47.456 [Warning][ (0)] [TID 6140][galaxy_client]: Failed to import owned games from platform 'steam', reason 'Aborted'. 2023-06-18 16:38:47.456 [Information][ (0)] [TID 6140][galaxy_client]: Discarding problem report because 5s elapsed since last sending of External-Platform-Operation-Fails problem. 2023-06-18 16:38:47.459 [Information][ (0)] [TID 6140][galaxy_client]: Scheduling problem report external_platform_operation_failed (type external_integrations_failed). 2023-06-18 16:38:47.459 [Information][ (0)] [TID 6140][galaxy_client]: Discarding the problem report, 'external_platform_operation_failed (type external_integrations_failed)', because 5s has elapsed since sending the last report. 2023-06-18 16:38:47.459 [Warning][ (0)] [TID 6140][galaxy_client]: Failed to import friends from platform 'steam', reason aborted

[don-de-marco]

Ah, frick. There's a typo in there. If you're willing to monkey patch the files, please replace packagie_id with package_id in protobuf_client.py.

I'll push a proper fix tomorrow

[engiefox]

No longer crashes, but nothing is syncing. The steam folder is empty and no games show on recent. Shows this in the logs: 2023-06-18 17:00:34,429 - steam_network.websocket_client - ERROR - Failed to establish authenticated WebSocket connection TypeError("'CMsgClientPICSProductInfoRequestPackageInfo' object cannot be interpreted as an integer") 2023-06-18 17:00:34,432 - steam_network.websocket_client - ERROR - Traceback (most recent call last): File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\websocket_client.py", line 111, in run await run_task File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol_client.py", line 113, in run await self._protobuf_client.run() File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 178, in run await self._process_packet(packet) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 518, in _process_packet await self._process_message(emsg, header, packet[8 + header_len:]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 525, in _process_message await self._process_multi(body) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 565, in _process_multi await self._process_packet(data[offset + size_bytes:offset + size_bytes + size]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 518, in _process_packet await self._process_message(emsg, header, packet[8 + header_len:]) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 535, in _process_message await self._process_license_list(body) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 672, in _process_license_list await self.license_import_handler(licensed_apps) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol_client.py", line 368, in _license_import_handler await self._protobuf_client.get_packages_info(packages_to_import) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 440, in get_packages_info await self._send(EMsg.ClientPICSProductInfoRequest, message) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\steam_network\protocol\protobuf_client.py", line 482, in _send body = bytes(message) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 629, in __bytes__ wraps=meta.wraps or "", File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 360, in _serialize_single value = _preprocess_single(proto_type, wraps, value) File "C:\Users\obliv\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\betterproto\__init__.py", line 346, in _preprocess_single return bytes(value) TypeError: 'CMsgClientPICSProductInfoRequestPackageInfo' object cannot be interpreted as an integer

[don-de-marco]

Oof. That's something I cannot fix from my phone. Apparently rebasing to the upstream version of this branch broke more than I anticipated. I'll get a fix out when I'm back from work tomorrow.

[don-de-marco]

@engiefox @Autistic-Unicorn please try now. my latest commit should fix the errors you reported

[Autistic-Unicorn]

It instantly started to import games into GOG Galaxy, now lets see if it gonna import them all and all other stuff

[Autistic-Unicorn]

There is lots of this error in logs. I think it shouldn't be like that

2023-06-19 17:13:25,085 - galaxy.api.jsonrpc - INFO - Handling request: id=497, method=start_game_times_import, params={'game_ids': ['108710']}
2023-06-19 17:13:25,085 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "495", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,085 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "496", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,085 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "497", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,202 - galaxy.api.jsonrpc - INFO - Handling request: id=498, method=start_achievements_import, params={'game_ids': ['256190']}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - INFO - Handling request: id=499, method=start_game_times_import, params={'game_ids': ['256190']}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - INFO - Handling request: id=500, method=start_achievements_import, params={'game_ids': ['207140']}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "498", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - INFO - Handling request: id=501, method=start_game_times_import, params={'game_ids': ['207140']}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "499", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - INFO - Handling request: id=502, method=start_achievements_import, params={'game_ids': ['35140']}
2023-06-19 17:13:25,203 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "500", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,204 - galaxy.api.jsonrpc - INFO - Handling request: id=503, method=start_game_times_import, params={'game_ids': ['35140']}
2023-06-19 17:13:25,204 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "501", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,204 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "502", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,204 - galaxy.api.jsonrpc - ERROR - Sending data: {"jsonrpc": "2.0", "id": "503", "error": {"code": 600, "message": "Import already in progress", "data": {"internal_type": "ImportInProgress"}}}
2023-06-19 17:13:25,302 - galaxy.api.jsonrpc - INFO - Handling request: id=504, method=start_achievements_import, params={'game_ids': ['354640']}
2023-06-19 17:13:25,302 - galaxy.api.jsonrpc - INFO - Handling request: id=505, method=start_game_times_import, params={'game_ids': ['354640']}
2023-06-19 17:13:25,302 - galaxy.api.jsonrpc - INFO - Handling request: id=506, method=start_achievements_import, params={'game_ids': ['2400']}

[don-de-marco]

That's "fine". Galaxy tries to import achievements while we're still importing the games list. The errors you see is just us telling Galaxy to wait until the import is done; eventually, these error messages should stop and achievement import continues/starts as normal

[ABaumher]

Lmk how it goes and I'll look to merge it in if successful.

[Autistic-Unicorn]

For now it only synced around 2500 games and doesn't do anything more than that. No game time imported, just constantly checks friends and nothing more in logs. It also crashed once without shoving anything wrong.

The logs don't show much. It first does a lot of "updating apps for package" and then it stops and nothing else happens (about importing games, there are logs about checking friends info which is still failing to upload into gog)

2023-06-19 19:04:07,891 - steam_network.caches.games_cache - DEBUG - updating apps for package 737478: [2064170]
2023-06-19 19:04:07,891 - steam_network.caches.games_cache - DEBUG - updating apps for package 729099: [1881700]
2023-06-19 19:04:07,891 - steam_network.caches.games_cache - DEBUG - 0 apps need to be imported: set()
2023-06-19 19:04:07,891 - steam_network.caches.games_cache - DEBUG - import queue status -- packages: 22 | apps: 0
2023-06-19 19:04:07,892 - steam_network.protocol.protobuf_client - DEBUG - Finished processing message Multi
2023-06-19 19:04:07,892 - steam_network.protocol.protobuf_client - DEBUG - Processing packet of 1492 bytes
2023-06-19 19:04:07,892 - steam_network.protocol.protobuf_client - INFO - [In] 1 -> EMsg.Multi
2023-06-19 19:04:07,892 - steam_network.protocol.protobuf_client - DEBUG - Processing message Multi
2023-06-19 19:04:07,893 - steam_network.protocol.protobuf_client - DEBUG - Processing packet of 4832 bytes
2023-06-19 19:04:07,893 - steam_network.protocol.protobuf_client - INFO - [In] 8904 -> EMsg.ClientPICSProductInfoResponse
2023-06-19 19:04:07,893 - steam_network.protocol.protobuf_client - DEBUG - Processing message ClientPICSProductInfoResponse
2023-06-19 19:04:07,894 - steam_network.protocol.protobuf_client - DEBUG - handling 0 apps
2023-06-19 19:04:07,894 - steam_network.caches.games_cache - INFO - importing 0 apps
2023-06-19 19:04:07,895 - steam_network.caches.games_cache - DEBUG - import queue status -- packages: 22 | apps: 0
2023-06-19 19:04:07,895 - steam_network.protocol.protobuf_client - DEBUG - handling 22 packages
2023-06-19 19:04:07,895 - steam_network.caches.games_cache - DEBUG - importing 22 packages
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 117651: [508720]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 892624: [2010240]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 892617: [2004340]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 892498: [1877190]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 892497: [1846510]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 892301: [2266370]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 885943: [2461220]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 885264: [2459460]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 884635: [2457970]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 883669: [2455180]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 881762: [2450030]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 881745: [2449980]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 880941: [2447720]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 875219: [2432010]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 874891: [2431310]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 874072: [2429250]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 871541: [2421830]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 870333: [2418290]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 868757: [2413660]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 838914: [2335720]
2023-06-19 19:04:07,896 - steam_network.caches.games_cache - DEBUG - updating apps for package 837937: [2333040]
2023-06-19 19:04:07,897 - steam_network.caches.games_cache - DEBUG - updating apps for package 780222: [2067610]
2023-06-19 19:04:07,897 - steam_network.caches.games_cache - DEBUG - 0 apps need to be imported: set()
2023-06-19 19:04:07,897 - steam_network.caches.games_cache - DEBUG - import queue status -- packages: 0 | apps: 0
2023-06-19 19:04:07,897 - steam_network.caches.games_cache - INFO - Setting state to ready
2023-06-19 19:04:07,897 - steam_network.protocol.protobuf_client - DEBUG - Finished processing message Multi
2023-06-19 19:04:09,118 - steam_network.protocol.protobuf_client - INFO - [Out] <EMsg.ClientHeartBeat: 703> (23B)

[don-de-marco]

I implemented a very aggressive caching mechanism: for all packages we only update those apps which we do not already have in our cache. That's where the line 2023-06-19 19:04:07,897 - steam_network.caches.games_cache - DEBUG - 0 apps need to be imported: set() comes from -- all the apps referenced by the packages are already known. However, I'd recommend to reset your steam_xxx.db file so that you get a clean start (you could also change the value of _VERSION in game_cache.py to a different value to achieve the same). I suspect that you got an incomplete cache because of the bugs we had before.

[engiefox]

@don-de-marco just ran with your new build, seems to run just fine. synced games from an existing cache and everything. about to run without a cache, one moment

[engiefox]

it hardly spent any time at all syncing with games already in the cache, definitely a lot nicer than waiting for it to finish on every startup.

[Autistic-Unicorn]

i deleted the .db file from C:\ProgramData\GOG.com\Galaxy\storage\plugins but the games are still listed in the app. What the heck?

[engiefox]

That happens if you don't use "Disconnect" for Steam first sometimes. Reconnect and then disconnect and it should work.

[don-de-marco]

i deleted the .db file from C:\ProgramData\GOG.com\Galaxy\storage\plugins but the games are still listed in the app. What the heck?

That's expected. Galaxy keeps an internal list of all games we imported so that it can show you your games without the actual plugin being up and running. If you want to get rid of that, click disconnect in the plugin menu and Galaxy will reset that information for you

[ABaumher]

I think this build might still have the no 2FA bug. I expressly recall testing that when first getting it working because i was sick of giving steam an email code. But apparently it didn't work, lol. If the fix works on the bugfix branch i can merge them both into the dev branch. If the no 2fa bugfix works I'll have to bump the release to 1.0.7 so we'll need to make sure all our alpha builds are at least 1.1

[engiefox]

Running without cache also works for games at least. Achieves and playtime aren't syncing, but I'm giving it a moment. edit: Ok, achieves/playtime don't seem to want to sync, but nothing obvious in the log from what I can read. GalaxyClient.log plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log

[don-de-marco]

(technical side note incoming) There's two databases in Galaxy: the one Galaxy itself uses and the one Galaxy provides for plugins to save data. So if you want to reset what Galaxy knows about the games coming from a given plugin, you need to disconnect the plugin so that Galaxy removes the information from it's own database. However, this does not remove or clear the information saved to the steam_xxx.db file - if you reconnect the plugin afterwards, our very own caches are still intact.

Now that I'm thinking of it, I might have caused an inconvenience with my current implementation. I should probably remove the _sent_to_galaxy flag from the cache as it now would prevent re-importing games after you disconnected the plugin. Removing the value from the cache makes the plugin re-import all games into Galaxy whenever the plugin starts but at least removes the stress on steam's end since we just pass on the information we already have in our cache

[Autistic-Unicorn]

Not sure why but it got timed out again. I tried logging in twice and same thing happens.

...
2023-06-19 19:38:54.065 [Information][ (0)] [TID 15860][galaxy_client]: Changed the bandwidth limit in the 'Game Download' processor to <NULL>.
2023-06-19 19:38:55.536 [Warning][ (0)] [TID 14984][galaxy_client]: Failed to import owned games from platform 'steam', reason 'Timeout'.
2023-06-19 19:38:55.537 [Information][ (0)] [TID 14984][galaxy_client]: Not sending problem report as the user has disabled it.
2023-06-19 19:38:55.542 [Information][ (0)] [TID 14984][galaxy_client]: Scheduling problem report external_platform_operation_failed (type external_integrations_failed).
2023-06-19 19:38:55.542 [Information][ (0)] [TID 14984][galaxy_client]: Not sending the problem report as the user has disabled sending reports.
...

...
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 1761270 (Half-Life: MMod); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 2334710 (I was born poor); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 1916920 (The Doll Beneath); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 1843190 (Brain Bait); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 1703930 (Kriegsfront Battlescaper); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - updating app 2424830 (篮球经理模拟器); parent: None
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - DEBUG - import queue status -- packages: 0 | apps: 0
2023-06-19 19:37:22,847 - steam_network.caches.games_cache - INFO - Setting state to ready
2023-06-19 19:37:22,847 - steam_network.protocol.protobuf_client - DEBUG - handling 0 packages
2023-06-19 19:37:22,848 - steam_network.caches.games_cache - DEBUG - importing 0 packages
2023-06-19 19:37:22,848 - steam_network.caches.games_cache - DEBUG - 0 apps need to be imported: set()
2023-06-19 19:37:22,848 - steam_network.caches.games_cache - DEBUG - import queue status -- packages: 0 | apps: 0
2023-06-19 19:37:22,848 - steam_network.protocol.protobuf_client - DEBUG - Finished processing message Multi
...

[Autistic-Unicorn]

when im gonna get some time im gonna get rid of all gog galaxy files and im gonna do a completely clean install. Maybe there is some issue on my side.

[don-de-marco]

I don't think the issue is on your end. Can you try the commit I just pushed? Maybe it's related to the aggressive caching I introduced

[Autistic-Unicorn]

After 3min it failed the import due to timeout. Happens every time i restart the gog. It looks like the plugin sees the games but none of them get imported into gog.

2023-06-19 21:10:10.855 [Warning][ (0)] [TID 14908][galaxy_client]: Failed to import owned games from platform 'steam', reason 'Timeout'.
2023-06-19 21:10:10.855 [Information][ (0)] [TID 14908][galaxy_client]: Did not find a value under the preference key 'previewBuildsEnabled' while reading.
2023-06-19 21:10:10.870 [Information][ (0)] [TID 14908][galaxy_client]: Scheduling problem report 'External-Platform-Operation-Fails': {"Arguments":{"applicationType":"GOG Galaxy","clientVersion":"2.0.67.2","detectedOsVersion":"win10","isRunningOnWine":false,"operation":"Owned games import - 2","os":"windows","osIdentification":"windows: Windows 10 10.0 (Build 19045)","platform":"steam","updateChannel":"master"},"ProblemReport":"External-Platform-Operation-Fails"}
2023-06-19 21:10:10.873 [Information][ (0)] [TID 14908][galaxy_client]: Scheduling problem report external_platform_operation_failed (type external_integrations_failed).
2023-06-19 21:10:10.883 [Information][ (0)] [TID 17040][galaxy_client]: Sending problem report of type External-Platform-Operation-Fails: {"Arguments":{"applicationType":"GOG Galaxy","clientVersion":"2.0.67.2","detectedOsVersion":"win10","isRunningOnWine":false,"operation":"Owned games import - 2","os":"windows","osIdentification":"windows: Windows 10 10.0 (Build 19045)","platform":"steam","updateChannel":"master"},"ProblemReport":"External-Platform-Operation-Fails"}
2023-06-19 21:10:11.075 [Information][ (0)] [TID 17308][galaxy_client]: Successfully sent problem report 'external_platform_operation_failed (type external_integrations_failed)'.
2023-06-19 21:10:11.106 [Information][ (0)] [TID 17040][galaxy_client]: The problem report was successfully sent.

[ABaumher]

@don-de-marco you may want to squash merge in the changes i had to make to the login process to fix the broken No 2FA option. I'll do a PR on this branch for you. That said, i'm merging it into better proto deps, so maybe it'll just throw up a conflict and i can fix it in merge.

Edit: engiefox may have found another bug as a result of fixing this. it won't appear for another ~6 months for the rest of the users, which is why i didn't catch it. i've made the user info cache a little more resistant to bad data, so it should be fixed. wait a moment.

[don-de-marco]

alrighty, while I was on it and since @engiefox kept reminding me about that nastily slow achievement import, I gave StatsCache a long-deserved touch-up as well. Everyone, please be so kind and give it a try. I want to see all the bugs dug out :'D

However, I don't think we can do much about the achievement imports for as long as we have that weird job_queue thing hanging around. From what I feel, that is where the import bottleneck is atm because it basically throttles the whole job queue down to 2 jobs being handled in parallel (that includes the whole send-message-and-wait-for-response round trip, btw). imo we should just send all requests off and asynchronically handle the responses similar to what we do with the PICSInfo message during license import

[don-de-marco]

I'm out for the night, just throws bugs etc at me. I'll work on them tomorrow

[ABaumher]

Actually tried firing them all off at once, it was monumentally slower, because it was blocking the recv calls until it finished. Now, sending one at a time and waiting for the recv properly might work, but i don't think the original dev know what run in executor was (the threaded version since this is io based)

[engiefox]

Only synced 6 achievements with the new build. plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log

[don-de-marco]

Mhm. The log looks fine, though. I see a whole bunch of achievements being imported. The only difference from what we did before is that I now provide an achievement id to Galaxy. I see two possible fixes for this. (a) find in src\backend_steam_network.py the line which says achievement_id=achievement.id_, and delete it/comment it (b) find in src\steam_network\protocol_client.py the line which says id_=32 * (achievement_block.achievement_id - 1) + i, and remove the - 1 so that it says id_=32 * (achievement_block.achievement_id) + i,

Please let me know if either of the solutions helped

[engiefox]

Ok, first one worked, tested twice. They appeared pretty quickly on startup before I deleted the .db to test from scratch. Definitely took less time than before, thank goodness.

[Autistic-Unicorn]

i have completely deleted gog and installed it again with only steam plugin installed. I'm still getting timeout error after about 3 minutes of opening the app. Nothing gets imported. Now im really not sure what is causing that.

[engiefox]

Should a different thread be made for the achievement thing? I don't want to clutter up the thread. This other problem sounds more serious.

[engiefox]

Tested the 2nd one and it didn't help. Still just the six achievements show.

[don-de-marco]

Thanks for testing @engiefox! Fix will be out shortly

@Autistic-Unicorn can you make an issue for this and attach some logs there? I feel this one needs more thorough investigation

[don-de-marco]

alrighty, one final set of changes. @engiefox @Autistic-Unicorn if you would be so kind and break it rq :)

[engiefox]

I don't seem to see any issues. What exactly should I expect?

[don-de-marco]

Hopefully exactly this :'D Do Achievements sync fine still?

[engiefox]

Yes, still the longest task, but it's tolerable. About 5 minutes from completing login to fully synced.

[engiefox]

Game import is probably the most impressive, less than 2 minutes to sync +800 games.

[don-de-marco]

Thanks. That one gave me the most headaches too :D

If @Autistic-Unicorn doesn't dig out some bugs during the afternoon, I'll declare this completed @ABaumher

[Autistic-Unicorn]

Probably won't gonna do anything more since im stuck with the same error over and over again. Did create an issue at https://github.com/ABaumher/galaxy-integration-steam/issues/33 but probably i should made it at your profile instead of ABaumher Sorry 😞

[engiefox]

Crash after running the plugin for a few hours, crash show on the client log ~line 658 13:31:26 plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log GalaxyClient.log

[ABaumher]

I'm hoping the bug engiefox found is just emblematic of our code still needing cleanup overall and not a bug in your code but i'll hold off until you give it a look.

I also merged versions 1.0.6 and 1.0.7 into better proto deps (as well as the batch file fix, which i'mo not pushing to Nebula because it'll break the approvals and i don't feel like redoing them. I don't believe there will be any errors as a result of this (no conflicts != no errors at runtime, ofc), but i may be wrong. I know for a fact that protobuf_client is conflicting now (as noted by the PR) but i'm not aware how bad of a conflict this is. Sorry for adding something to your workload, lol.

[don-de-marco]

@engiefox honestly, I think you just have too many games installed. From your log I can see that Galaxy wants to import ~800 games. After exactly 20 seconds (and about 650 games done), Galaxy cuts us off because it took too long. I don't see any opportunities on how to improve our code, it's pretty straight-forward already. There's one single thing we could maybe change (but I'm not in favor of that idea): currently, we parse the state file of each game and check whether it's currently being downloaded (and then return the in-flight file size of what's currently already there). Instead, we could take the library information file and always return the SizeOnDisk element regardless of the game maybe being currently downloaded. That's the only thing I see that would net us some time. But as I said, I don't really like that idea and would rather have Gog just increase the import timeout to 30s or even 60s.

[don-de-marco]

@ABaumher I rebased this branch onto your current better_protobuf_deps. I don't see anything obvious going wrong so I guess we have a clean state rn