Evaluate possibility of "inventory changes" for ASFB

[JustArchi]

Checklist

• [X] I read and understood ASF's Contributing guidelines
• [X] I also read Setting-up and FAQ, I don't need help, this is an enhancement idea
• [X] My idea doesn't duplicate existing ASF functionality described on the wiki
• [X] I believe that my idea falls into ASF's scope and should be offered as part of ASF built-in functionality
• [X] My idea doesn't violate the Steam Subscriber Agreement
• [X] My idea doesn't violate the Steam Online Conduct
• [X] This is not ASF-ui suggestion

Enhancement purpose

Right now ASF sends whole inventory state to ASFB, which is beneficial in a lot of aspects as we're always sure that we have the correct state. However, some people have really huge inventories which puts a lot of load on the server (and their links as well).

Solution

It could be a good idea to introduce "inventory diff" and announce that, it'd work like this:

• Initially you announce in full, server will respond with inventory version that ASF saves in the bot's database along inventory state.
• On each change in inventory, we fetch it and compare with the saved state. If there are no differences, we skip announce entirely, otherwise we transmit the differences - additions and deletions, along with the inventory version we made it against.
• Server will check version against what it saved, if it's different (something went out of sync), we'll respond with some kind of conflict and ASF will announce new state as usual. Otherwise, server will accept the diff, update the database, then respond with new inventory version.

Why currently available solutions are not sufficient?

It's working good enough, but with further increase in amount of users, inventories and items, we might reach a point where transmitting 50+ MB of (already compressed) data is not sufficient for operation.

Can you help us with this enhancement idea?

Yes, I can code the solution myself and send a pull request

Additional info

I'm not sure what to do about potential desyncs. We could have a situation where cache corruption or something else happens and we'll have outdated inventory state. Perhaps we could do bare minimum and along version send at least number of items we have right now, so server could do some bare minimum of state checking - we still risk corruption when we lose A and gain some other B, but that should be far less likely and resolve itself in time.

Other suggestions welcome.

[JustArchi]

Another idea: we could implement some algorithm for calculating "checksum" of inventory state. This way we wouldn't need to keep state in the cache even, only saved checksum. The question is how that algorithm should work...

[JustArchi]

Bad idea, checksum alone won't do, BUT together with saving inventory in the cache, we can actually make it work.

Assuming inventory state is generated by some deterministic algorithm, ASF has previous inventory state, and current one. If it's determined that they're different, ASF sends previous inventory state, inventory changes, and newly calculated new inventory state. ASF backend can check if previous inventory state is the same in DB, if yes, it can add inventory changes and calculate its own new inventory state, then compare it with the one reported by ASF. If it's the same, we have 100% guarantee we didn't lose anything and there is no desync. If anything doesn't match (previous state, or new one), we tell ASF conflict and it needs to do full announce.

I don't see a problem anymore, I only need deterministic algorithm for inventory state now.