CJBuchel
commented
1 day ago Local first is an issue with this. My current setup makes it fairly easy to poll the server for data, and then switch to local storage as a backup. Because all the data I send, receive and store is hashmaps and itemised structures in that hashmap. So any updates I get I can immediately store them in the local version of the hashmap.
And then any of my "mutations" which access the db/local storage just grab the data like normal. A completely synchronised tree. With GraphQL based systems, because this also handles the "mutation" on the query side, rather than me polling directly for the exact data. It makes it more difficult to "store" or "cache" these items.
Because the data I'm getting back from the server won't be the table, it will be specific data queried from that table. Which means any offline systems would need to be handled by the client in a different manner. Likely a bit more manual.
But, in truth. This is how it should be anyway. Most people who will look at a server, and see that it uses GraphQL + SQLite, will just query like normal to it. If the client uses an offline version, it should be up to the client to implement that type of logic to handle it. Rather than forcing everyone to use a specific style, such as hashmap pub sub for the sake of offline storage. When not everyone will be using offline storage.
A bit more thought will need to be put into this, either I can do the manual method. I.e, "if user goes onto scoreboard, query for team names, rank and scores, then store those exact results locally in a hashmap for next time". Or caching method. I.e "if user goes onto scoreboard, graphql query for items and have that same query system cache the response somehow locally, by tagging and stringifying the results?"
The latter is more complex functionaly, but does automate the process for every query to be cached. Which is a double edged sword. In reality we don't need to store every bit of data locally. Just the key things, so I'm aiming for the former over the latter. As that seems to more be the standard anyway for these types of applications.
Is your feature request related to a problem? Please describe.
TMS uses a very specialised network system, with a hashmap KVDB that provides updates in realtime using custom socket protocols.
The system works great for instant messages and fast data updates. Including synchronising using the custom checksum based system, to verify the data between client and server.
However, it's extremely complex. As time progresses it will be more and more difficult to update, and change the type of network because of how non-standard it is.
Not only this, but because the network was designed as "library first" in mind, the database is it's own separate network, with it's own set of HTTP and WS connections. This is fine, but considering the database should not be a library at all, it causes confusion and isn't great to work with.
When TMS becomes more documented, the API should be exposed to users to also build their own plugins and add-ons for an event, and this custom solution is very difficult to work with outside of the TMS scope.
Describe the solution you'd like
Switch to a more standardised networking solution. This is a difficult task to undertake not only because TMS is very network reliant heavy. But also because there is no system that works for our needs that is perfectly standardised.
We need a local first storage solution, coupled to a realtime synchronising network that has a flexible query system with pub sub streaming. Almost all realtime updating data sources are custom to an extent, which makes this a difficult thing to "standardise".
One standard which almost fits all of these categories is GraphQL which is primarily a query language like SQL, but much more flexible. Especially useful for my needs, where only certain bits of data are necessary on each page. I.e, Matches hold the team number, but not the team. So I have to get the map of both matches and teams to get one single match with the team name. GraphQL resolves this with specific queries that "zip" types together from the database. And best of all, it defines a subscription standard which is already incorperated into some libraries https://graphql.org/blog/2015-10-16-subscriptions/
Coupled with SQLite as the primary database, and the system becomes functional, modern and standarised. So external users can also modify and work with it easier. The only downside, is that GraphQL is just a standard, and while it does have subscription standards. The way it's implemented is left to interpretation, for a rust + flutter based system, it might be difficult to put all this together, and then replace the existing hashmap pub sub system I currently have.
But it might be worth it, the biggest problem is this will take time. Lots of it, 6 months minimum to replace the current implementation reliably.
Describe alternatives you've considered
Redis, EchoTree (Staying with the Current), custom pub sub using websockets.
Additional context
No response