FriendsOfGalaxy / galaxy-integration-steam

Integration with Steam for Galaxy
374 stars 76 forks source link

Occasional but consistent: Connection Lost (previously: "Offline. Retry") #74

Open ryanpcmcquen opened 4 years ago

ryanpcmcquen commented 4 years ago

This has happened consistently to me with the Steam integration. Clicking 'Retry' does nothing. The only fix seems to be logging in with Steam again using the plugin.

Logs: plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log


EDIT by FriendsOfGalaxy

Problem Steam backend responds for ClientLogon message with InvalidPassword (EResult==5) when using login key (token) (so for example when you restart Galaxy).

In the past it was known to happen that already "invalid" token works OK a few hours later. That is why current workaround is to switch to "Offline" instead of logging the user out.

https://steamerrors.com/5 explains that InvalidPassword is returned when:

Returned when you provide an incorrect password or possibly some other kind of authentication token. This can also be returned in response to a CM logon if your network is being temporarily login rate-limited.

Reproduction Not clear, but one of most reliable ways to reproduce the problem is with 2 machines:

Ideas to fix this

jimbo2150 commented 4 years ago

This happens a lot here too. Only fix I've found is to disconnect and reconnect Steam from Galaxy.

ReiTung commented 4 years ago

Guess this is currently THE most common issue with the Steam plugin, also added by others under different issue# here. Also, if you look into the Galaxy Bugtracker itself, you will find tons of these entries as well.

Seems to be a nasty one (I am not a scripting guy at all) - really looking forward to get a sustainable fix for it. The Devs mentioned in another thread that until then the only currently available mitigation is to disconnect and reconnect the plugin, yet with all disadvantages ...

Thanks for all your effort! ReiTung

dandepeched commented 4 years ago

I had this issue and fixed it with disconnect and reconnect. Now I have frequent error of plugin crash - #78

ltegman commented 4 years ago

Same here. Looked into it a little bit and it appears that after a while when the plugin tries to authenticate the token with Steam it will constantly fail with a "Invalid Password" error. I assume this means whatever token this app is storing for authentication eventually expires and they need some sort of mechanism for refreshing tokens without needing you to constantly logout and log back in. Don't have the time to look into this at the moment, but I'd like to eventually. Hopefully someone will get to it before me though!

Funsponge commented 4 years ago

Same here. Looked into it a little bit and it appears that after a while when the plugin tries to authenticate the token with Steam it will constantly fail with a "Invalid Password" error. I assume this means whatever token this app is storing for authentication eventually expires and they need some sort of mechanism for refreshing tokens without needing you to constantly logout and log back in. Don't have the time to look into this at the moment, but I'd like to eventually. Hopefully someone will get to it before me though!

If there was just a way to easily reset this authentication token without disconnecting steam and ripping everything out of the library (which in turn resets all recently added and played statistics) it wouldn't be so frustrating.

ReiTung commented 4 years ago

While I hoped this issue might have been mitigated by the latest v0.52, it still seems to exist ... Just had the old behavior with the plugin going into offline/retry state simply when not using Galaxy for a few days ... I don't think it has to do with multiple computers and such as it times out (?) Also simply when not using it for a few days. Without being a coder at all, looks more as if the steam token gets outdated or such.

FriendsOfGalaxy commented 4 years ago

Just an update on this:

@ltegman if you have time you may try to check out https://steamcommunity.com/chat:

ReiTung commented 4 years ago

If there is anything Non-Coders can do to support/test, please let us know. Thanks for all your committment

FriendsOfGalaxy commented 4 years ago

I've edited OP to add more background for anyone who can help.

@ReiTung Non-Coders are welcomed for testing whenever we code something useful, stay tuned :)

kyuuketsukikurai commented 4 years ago

Still having this. Always happy to help however I can.

MarcoBrieden commented 4 years ago

@FriendsOfGalaxy Which exact API are you using here? Is there any documentation available at all?

When you log in do you only get the one token you use for login or any additional ones? I'm wondering if the API uses refresh tokens to renew expired access tokens.

jimkeir commented 4 years ago

Admit I've not looked at the code but yes, I've used other auth systems that differentiate between access and refresh tokens. You retain access for a fixed period and after that have to handle the failure by requesting a new access code using a refresh code that's provided with the initial login.

mikenerone commented 4 years ago

I'm a complete neophyte in terms of what APIs Steam may have available, but the token handling described above sounds very much like OAuth 2, the most common token-based authentication protocol. Along with the fact that at least one API is documented as using OAuth 2, I suspect that applies here.

For now, if the plugin's Retry link could simply be made to bring up a new login window to allow it to reconnect, that would be far less impactful for users than having to disconnect and lose all metadata. Right now, the link seems to do absolutely nothing.

SparrowBrain commented 4 years ago

I've tried looking into the issue, but it's a bit hard to read the code coming from the outside. @FriendsOfGalaxy could we remove the deprecated code marked with TODO's to reduce the complexity? Is it still doing anything?

On a similar note, does anyone know a good way to test/invoke the plugin outside of GoG Galaxy? I presume calling authenticate() using PyCharm will work, but haven't tried it yet.

SparrowBrain commented 4 years ago

So I was debugging the plugin while attached to process that launches it with PyCharm, and noticed that user_info_cache.UserInfoCache._check_initialized() was getting called after the TimeoutError in the authenticate() method happened.

There's probably some race condition, it warrants deeper digging. For the moment, I think it's worth pushing the workaround, if it helps people to retain their Steam integration.

EDIT: Sorry, wrong issue for the pull request.

MarcoBrieden commented 4 years ago

@FriendsOfGalaxy (@SparrowBrain ?) can you high level run us through how the code is doing the login (with pointers to the file/function)? Something like: Get initial token (file.py/get_token) Check if token is available (file.py/load_token) Check if token is expired (file.py/check_token) Renew token (file.py/renew_token) etc.

I'd love to help resolve this issue.

FriendsOfGalaxy commented 4 years ago

There is no refresh token flow here. We're using STEAM internal protobuf communications (by websockets).

So we're theoretically following Steam desktop client flow. The most known interface implementation is here: https://github.com/SteamRE/SteamKit. There is also project in Python for this https://github.com/ValvePython/steam (client part). Differences between this and our implementation are:

I'm not aware of any Steam protocol documentation except for this unofficial one: https://steamerrors.com/.

Plugin code login flow:

(Edited) So once Steam network loop is running we send authentication message and then wait for Steam response https://github.com/FriendsOfGalaxy/galaxy-integration-steam/blob/master/src/protocol/protocol_client.py#L192 Meanwhile the main code (plugin.py) is waiting until plugin._user_info_cache.initialized Event is set. This Event is set whenever we receive any part of user_info_cache (token, username, sentry_sha) AND all parts are currently in the cache. Logic check to set initialized Event is here: https://github.com/FriendsOfGalaxy/galaxy-integration-steam/blob/master/src/user_info_cache.py#L18.

In practice, we're waiting for successful ClientLogOnResponse (https://github.com/FriendsOfGalaxy/galaxy-integration-steam/blob/master/src/protocol/protobuf_client.py#L324) OR timeout (currently 30s). Tech debt: we should stop on error received from logging method; currently this is not propagated properly and the code is waiting until timeout is hit.

For this issue bug, we probably should be interested in protocol_client.py::authenticate_token. We're waiting for login future there that is set on LogOnResponse from Steam that I've mentioned above. We've got EResult.InvalidPassword there (Steam client does not receive it so it should be doable). And we have it as a response on ClientLogOn Steam message created and sent in log_on_token method from protobuf_client.py: https://github.com/FriendsOfGalaxy/galaxy-integration-steam/blob/master/src/protocol/protobuf_client.py#L155

Input for this Steam ClientLogon message contains:

There are more things that Steam is using (screenshot from Nethook): nethook-logon-example I've already mocked all of them once, but it didn't solve the issue. Maybe I was mistaken and we should go through it once again.

Both login_key and sentry file can be changed on steam backend request. We're waiting for those messages and updates login_key or sentry in UserInfoCache and cache it using store_credentials API method whenever any of its elements (username, token sentry_sha, ...) has changed (see plugin.py tick method that is called every 1 second). _Tech debt note: this should probably be the only place where store_credentials method is called._

Maybe the solution lies in handling different messages? Maybe the is wonky bug in the code (it is alreayd a bit complicated)? Maybe Steam authorize itself in the second place we miss and then it does not receive "Invalidpassword"? Maybe we ignore important message? I don't know.


could we remove the deprecated code marked with TODO's to reduce the complexity? Is it still doing anything?

@SparrowBrain yes, we can do it. That was for migration purposes a few months ago and probably most of existing users already re-login Steam plugin at least once.

SparrowBrain commented 4 years ago

@FriendsOfGalaxy thanks for the info on how it works. I would definitely love to simplify it - async should allow us to write more sync looking code. I hope I'll be able to help, once I understand what's happening better.

So far I haven't managed to reproduce the 'error 5'.

Petricora commented 4 years ago

I can confirm that I managed to resolve this by following these instructions from user Veritas.Deus on the GoG forums and deleting two sets of files.

List of steps:

  1. Open GoG Galaxy, disconnect the Steam integration, and close GoG Galaxy.
  2. Open %localappdata%\GOG.com\Galaxy\plugins\installed\ and delete the "steam" folder.
  3. Open %programdata%\GOG.com\Galaxy\storage\plugins\ and delete the "steam" file.
  4. Open GoG Galaxy and initiate Steam integration. After the plugin is downloaded log in and provide the Steam Guard code. In my case Galaxy connected almost immediately.

I realise that this is a work-around but perhaps it will provide some clue as to why the steam plugin goes offline.

kyuuketsukikurai commented 4 years ago

I would think that's just an even more extreme version of disconnecting the account?

Funsponge commented 4 years ago

I can confirm that I managed to resolve this by following these instructions from user Veritas.Deus on the GoG forums and deleting two sets of files.

List of steps:

1. Open GoG Galaxy, disconnect the Steam integration, and close GoG Galaxy.

2. Open  `%localappdata%\GOG.com\Galaxy\plugins\installed\` and delete the "steam" folder.

3. Open  `%programdata%\GOG.com\Galaxy\storage\plugins\` and delete the "steam" file.

4. Open GoG Galaxy and initiate Steam integration. After the plugin is downloaded log in and provide the Steam Guard code. In my case Galaxy connected almost immediately.

I realise that this is a work-around but perhaps it will provide some clue as to why the steam plugin goes offline.

You're deleting the plugin and its data entirely, but it will still disconnect again when the auth expires.

Petricora commented 4 years ago

I would think that's just an even more extreme version of disconnecting the account?

You're deleting the plugin and its data entirely, but it will still disconnect again when the auth expires.

This is why it's a work-around, not a fix. On all previous occasions, after the auth expires, the plugin didn't fully disconnect but remained in the "offline" status (with Steam games displayed in Galaxy), until I'd manually disconnect it.

For the time being this at least allows for a reliable method of reconnecting the plugin. Game tags in Galaxy are unaffected by this procedure.

FriendsOfGalaxy commented 4 years ago

@Petricora I can make the plugin disconnect on eresult 5 again. I've hoped making it offline will give people ability to play installed games, but there is apparently a bug in Galaxy that makes it impossible. Second reason was it sometimes happen that eresult 5 is temporal problem. I was albe to log in with the same login_key after some time.

But for now I see that people tends to disconnect it anyway, so it probably more annoying for them I would "Lost Connection" itself

Varstahl commented 4 years ago

3. Open %programdata%\GOG.com\Galaxy\storage\plugins\ and delete the "steam" file.

This. Made it painless to reconnect and update, and while at it I've installed the zip version linked in #81 for good measure. Everything works so far, we'll see.

FriendsOfGalaxy commented 4 years ago

As it helped for #81 case, I'm reposting testing build here:

Guys, please try out this build (windows only): unique_ip.zip This build adds custom obfuscated_private_ip for LogOn message (I've just used hash of computer MAC address). Works for me - I'm no longer kicked off from a registered session in case when simultaneously logged in plugin on 2 machines.

To install:

ghost commented 4 years ago

@FriendsOfGalaxy I still get eresult 5 with your provided version.

kyuuketsukikurai commented 4 years ago

As it helped for #81 case, I'm reposting testing build here:

Guys, please try out this build (windows only): unique_ip.zip This build adds custom obfuscated_private_ip for LogOn message (I've just used hash of computer MAC address). Works for me - I'm no longer kicked off from a registered session in case when simultaneously logged in plugin on 2 machines.

To install:

  • close Galaxy
  • unpack this zip to %localappdata%\GOG.com\Galaxy\plugins\installed\
  • remove previous steam build from %localappdata%\GOG.com\Galaxy\plugins\installed\ (usually folder name starts with steam_...)
  • relaunch Galaxy

Just to confirm, we're supposed to unpack the zip, creating a folder called "unique IP" in the plugins, then clear out the Steam folder? Or is this meant to be put into the Steam folder?

FriendsOfGalaxy commented 4 years ago

@kyuuketsukikurai just make sure final structure contains:

jimkeir commented 4 years ago

This worked fine until this morning. One oddity; I powered up my laptop last night and it requested SteamGuard authentication despite being used about ten days ago, before GOG was reconnected with the version above. The main PC was off at the time. On switching that on this morning (with the laptop off), GOG lost the Steam connection. Steam itself connected immediately. I'll re-try this later and get a network capture of Steam starting up if the same happens.

dhowland commented 4 years ago

I have this exact same issue

dhowland commented 4 years ago

I read through the thread again. Is there no way to get the plugin to re-authenticate without disconnecting and purging all game data?

kyuuketsukikurai commented 4 years ago

As it helped for #81 case, I'm reposting testing build here:

Guys, please try out this build (windows only): unique_ip.zip This build adds custom obfuscated_private_ip for LogOn message (I've just used hash of computer MAC address). Works for me - I'm no longer kicked off from a registered session in case when simultaneously logged in plugin on 2 machines.

To install:

  • close Galaxy
  • unpack this zip to %localappdata%\GOG.com\Galaxy\plugins\installed\
  • remove previous steam build from %localappdata%\GOG.com\Galaxy\plugins\installed\ (usually folder name starts with steam_...)
  • relaunch Galaxy

I still have the issue with this version, but it seems to happen less often.

Varstahl commented 4 years ago

@dhowland yes, there is. You just need to do this and reconnect https://github.com/FriendsOfGalaxy/galaxy-integration-steam/issues/74#issuecomment-706574298.

Also, I've replaced the steam plugin content with the uniqueid one (didn't create a different directory), and it's been working fine for the last 4 weeks or so. At some point the steam and epic plugins error'd out, but they reconnected fine, so I'm guessing it was a problem on my end rather than the plugin itself.

kevinff commented 3 years ago

Tried twice and twice the next day i had to disconnect, so didn't help for me. But don't give up!

ReiTung commented 3 years ago

Hi, so, for whatever reason the issue did not happen for quite a long time for me. In the past, Access Denied (5) was thrown back after 2 or 3 days at the latest, and no way to bring the plugin back online without Disconnecting/Reconnecting. I decided to do some explicit testing and therefore deleted the logs between some of the steps to make them comparable to you; find them below (all with v0.53 for now). However, over the last period it took several weeks until Error 5 came up again (note I am not playing each day, but it was considerably longer working than in the past) today.

What I believe might have been different again (I need to explicitly re-run that) is that I was online in Steam when I started GOG (updated to v2.0.30.20 in the meanwhile) on the same machine. - before, when I have not been on Steam at the same time, it still allowed to connect. Now, not sure if that could be a "smoking gun" but maybe Steam returns an Access Denied and declares the token invalid if it gets a second login attempt from a different application - maybe there is some kind of mechanism on Steam side checking that. Once the plugin went into Offline Mode, no way to reanimate as usual.

I will redo that with the "unique-ip" version of the plugin, but honestly think that this will not make a difference.

thanks ReiTung

logs_with_offline_v053_after_fresh_install.zip logs_with_online_v053_after_fresh_install.zip

logs_with_offline_v053_before_fresh_install.zip logs_with_crashed_v053_before_fresh_install.zip

Varstahl commented 3 years ago

That's sort of what happened, because without different login tokens it all looks like a same (re)connection, so it (correctly) invalidates the old one when a new one is created. As I said, replacing the contents of the steam plugin with the unique-ip one let me login correctly with no issues, regardless of whether steam is launched first or last, same machine or not. Been running fine like this for the past 2 months or so.

ReiTung commented 3 years ago

I will redo the tests with the unique-ip version these days. Fingers crossed.

kevinff commented 3 years ago

Hi, so, for whatever reason the issue did not happen for quite a long time for me. In the past, Access Denied (5) was thrown back after 2 or 3 days at the latest, and no way to bring the plugin back online without Disconnecting/Reconnecting. I decided to do some explicit testing and therefore deleted the logs between some of the steps to make them comparable to you; find them below (all with v0.53 for now). However, over the last period it took several weeks until Error 5 came up again (note I am not playing each day, but it was considerably longer working than in the past) today.

What I believe might have been different again (I need to explicitly re-run that) is that I was online in Steam when I started GOG (updated to v2.0.30.20 in the meanwhile) on the same machine. - before, when I have not been on Steam at the same time, it still allowed to connect. Now, not sure if that could be a "smoking gun" but maybe Steam returns an Access Denied and declares the token invalid if it gets a second login attempt from a different application - maybe there is some kind of mechanism on Steam side checking that. Once the plugin went into Offline Mode, no way to reanimate as usual.

I will redo that with the "unique-ip" version of the plugin, but honestly think that this will not make a difference.

thanks ReiTung

logs_with_offline_v053_after_fresh_install.zip logs_with_online_v053_after_fresh_install.zip

logs_with_offline_v053_before_fresh_install.zip logs_with_crashed_v053_before_fresh_install.zip

I agree there might have something related to connecting to steam and/or connecting to the web chat.

I have a steam plugin for pidgin, and i noticed everytime i open the chat in the steam site, it disconnects/reconnects me to steam on pidgin. Not sure if related, but there might have some things to reset after connecting to steam or the steam web chat.

But how are doing the steam clients, the mobile app, and softwares like playnite? Maybe there's a token to refresh?

SlyAceZeta commented 3 years ago

This issue is so obnoxious 😭 It's always happening to me, none of the recommended solutions have helped.

ReiTung commented 3 years ago

Guys,

unfortunately bad news ... issue is still there (happened after 2 or 3 days) with the UNIQUE_IP fix ... same test scenario as above, i.e. Steam Client and Galaxy running and opened at the same client. :(

Logs attached.

thanks for digging in further.

ReiTung

logs_with_online_UIP_after_fresh_install.zip logs_with_offline_UIP_after_fresh_install.zip

Laserstrahl commented 3 years ago

Same here. Keep getting the issue, I tried all the fixes, they only help for short time. I also use multiple PCs with Steam (not with GOG so). So unfortunately Steam integration with GOG Galaxy is not realy useable for me. Would be great if the GOG guys would help the devs here out. I mean they are advertising that plugin too.

josh-loveholidays commented 3 years ago

@Laserstrahl same here really. I've switched to Playnite in the mean time.

SlyAceZeta commented 3 years ago

Can't we get some eyes on this @FriendsOfGalaxy? Steam integration is one of the big ones for GOG Galaxy, and this plugin is not functional if it keeps crashing every time we relaunch the program. This bug has been here for months, can't we get anyone with WebSocket knowledge to look into this?

SparrowBrain commented 3 years ago

Hi Azekthi. The issue for me was the way the plugin caches steam servers. I fixed it for myself by temporarily disabling the cache. If you want to try, you could:

  1. Go to file %LOCALAPPDATA%\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\servers_cache.py
  2. Make a backup of the file (just in case).
  3. Add a new line after line 30 - return dict()
    def _read_cache(self):
        return dict()
        cache = json.loads(self._persistent_cache.get('servers_cache', 'null'))
  4. Restart GoG Galaxy
  5. See if that fixes it for you
  6. You can revert the old file or delete the line you added (it should load your library faster without the modification).

See if this helps you. I had the ambition to help fix these issues in the plugin, but there's so much fixing to do, I don't know where to start, so kinda gave up...

SlyAceZeta commented 3 years ago

@SparrowBrain That didn't help. :/

Laserstrahl commented 3 years ago

I tried this workaround, which leads for me to a permanent crash of the plugin. Actually the UI tells mit it is crashed. I have no Idea where to find the log files (googles doesn't help...) otherwise I'd try to check them.

I find it strange that it crashes, since your quick hack should do the same as when the cash would be empty (Line 34 in that file).

SparrowBrain commented 3 years ago

Bugger. I guess this will be the socket issue, that one I wasn't able to reproduce, so no idea what's happening there. Sorry..

@Laserstrahl GoG Galaxy logs are here: %PROGRAMDATA%\GOG.com\Galaxy\logs\ Most recent file will be plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log

Laserstrahl commented 3 years ago

@SparrowBrain Thanks for advice, the crash came from a wrong tabulator. Fixed it. But the workaround isn't helping. Same issue as before:

The Log looks the same with or without your suggested modification.

_2021-01-01 18:44:24,014 - galaxy.api.jsonrpc - INFO - Sending notification: method=push_cache, params={'data': '****'} 2021-01-01 18:44:24,017 - galaxy.api.jsonrpc - DEBUG - Sending 695444 bytes of data 2021-01-01 18:44:24,135 - protocol.protobuf_client - INFO - Sentry present 2021-01-01 18:44:24,135 - protocol.protobuf_client - INFO - Sending log on message using token 2021-01-01 18:44:24,135 - protocol.protobuf_client - INFO - [Out] <EMsg.ClientLogon: 5514> (84B) 2021-01-01 18:44:24,338 - protocol.protobuf_client - INFO - [In] 1 -> EMsg.Multi 2021-01-01 18:44:24,338 - protocol.protobuf_client - INFO - New session id: -247612680 2021-01-01 18:44:24,338 - protocol.protobuf_client - INFO - [In] 751 -> EMsg.ClientLogOnResponse 2021-01-01 18:44:24,338 - protocol.protobuf_client - INFO - Failed to log on, reason : eresult: 5 eresult_extended: 5 count_loginfailures_to_migrate: 0 count_disconnects_to_migrate: 0

2021-01-01 18:44:24,339 - protocol.websocket_client - ERROR - Failed to establish authenticated WebSocket connection, retrying after 20 seconds Traceback (most recent call last): File "C:\Users\Laserstrahl\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\protocol\websocket_client.py", line 72, in run await auth_task File "C:\Users\Laserstrahl\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\protocol\websocket_client.py", line 230, in _authenticate raise e File "C:\Users\Laserstrahl\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\protocol\websocket_client.py", line 208, in _authenticate ret_code = await self._protocol_client.authenticate_token(self._user_info_cache.steam_id, self._user_info_cache.account_username, self._user_info_cache.token, auth_lost_handler) File "C:\Users\Laserstrahl\AppData\Local\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8\protocol\protocol_client.py", line 200, in authenticatetoken raise galaxy.api.errors.BackendError({"result": result}) galaxy.api.errors.BackendError

phtran72 commented 3 years ago

I tried disabling the server cache in the regular steam plugin and that didn't help. However, disabling the server cache in the unique_ip plugin (from the posting of Oct 13, 2020 in this thread) seems to fix my disconnect issues. The unique_ip mod alone reduced the frequency of disconnects, but they still occurred if I jumped between computers often enough, but unique_ip w/ server cache disabled works even better. I've been running it for a few days now, hopping between my laptop and desktop frequently, logging out then logging back in several times a day, and no disconnects so far. Previously, even with unique_ip alone, that many login cycles between my two computers would have certainly caused a disconnect. (And the standard plugin would have disconnected after a single login jump between my computers.)

Hope this sheds some light on the problem.

Laserstrahl commented 3 years ago

@phtran72 I tried it, still getting "2021-01-03 14:19:33,568 - protocol.protobuf_client - INFO - Failed to log on, reason : eresult: 5"

And by the way, I have the same scenario, 2 PCs which are both connected to one steam account (desktop pc and HTPC for the living room)