Server JSON is too big, split it to smaller files

[TomLewis]

So, I got my JSON export working with the latest update! that's good news! Slight issue, its all in one giant 283MB file haha, can't be pulling that client side easily.

Can I suggest exporting JSON the same way you do the current structure for players, /export/json/server/player/$username or $UUID/$username or $UUID.json

JSON is designed to be a super lightweight method to send data to a client, so you only load what's necessary, so in my case, I will have a user page, which I will be pulling the plan JSON from, the JSON folder is set to a public location, this way the JSON can be loaded via simple Javascript client call, at present its 1 giant 283MB file, which is a bit of a large file to pull.

You can definitely parse this from another scripting language first, but that defeats the purpose, I would just do an SQL query!

Same for general server JSON; /export/json/server/$servername.json

Thanks!

[AuroraLS3]

Yeah, I guess the server.json is not really good.

Why is it so big?

In the original request for json only player.json was considered, so server.json was an extra.

The player json was implemented by running the java object Plan uses through Google Gson (JSON parsing library).

Since both player and server use similar object it was easy to use the same thing to run the server object through Gson, resulting in this big blob - It includes all data Plan uses while calculating the values for /server page

---

Since both are dependent on the underlying implementation of the java class, they might change from version to version. 5.0.0 will include a redesign of the pages, during which the server object will go through drastic changes.

---

I'll call this issue "Re-implement Server JSON". It's going on the pile that is dealt with after/during 5.0.0 which is still quite far away.

In the meanwhile, a short term solution might be throwing the server.json on a GraphQL server, Apollo is a good one, although it requires knowledge of node.js https://github.com/apollographql/apollo-server

[AuroraLS3]

Since you might start relying on server json, I have to give this warning: It's going to change a lot.

Here is the redesign for the server page in case you have not seen it https://rsl1122.github.io/Plan-page-preview/server.html

I plan on storing as little as possible in memory and doing a lot of SQL for each number on the page, so some of the things on the current server json will likely not exist after the redesign is implemented.

The player json will likely remain very similar, as the player page does not require large changes.

[AuroraLS3]

After re-reading what you wrote I think it is possible to do /server/playername.json instead of a single big blob quite easily.

[TomLewis]

Nice redesign! It shouldn't be too difficult to update any changes to structure, It takes it from needing SQL queries in a scripting language like PHP to being pure client side loading in async, which also means you can host it on a static site on github pages or netlify.

im only just getting started with JSON/Javascript, I've been using PHP for 10+ years and im loving loading data live in the client via JSON. This will also make it easier for people to make third party apps, who knows maybe we could throw something together in electronjs.org for a cross platform desktop app for live monitoring later on! (Future project idea I wanted to do!)

This is acting a bit like an API, except there are no restrictions haha I guess the "proper" way to do it would be to serve the JSON calls live from your web-server generated live, so you don't have all those files, I have my own micro PHP API call for my own server, that just processes SQL lookups and spits out JSON. (contradicting myself here since I don't use the web server!)

The one thing that really bugs me with the current web-design is the crazy load times for the page to spin up, it should all be instant with the generated HTML, takes 3x as long on mobile too.

If you need any help, im happy to help with the web frontend stuff.

Edit: that night mode is gorgeous, I would want that as my main theme.

[AuroraLS3]

About loading json for the numbers

Yes front-end rendering is nice, but I dislike javascript (and web-dev mentality of jumping to the next hot framework of the month).

Vanilla JS, jQuery & Modern JS (EcmaScript) all come with their own flavor of problems. With vanilla JS DOM modification is difficult. With Modern JS comes npm & polyfills. jQuery is not very compatible with some frontend frameworks such as React.

Plan uses jQuery for some things.

Why the current website appears slow

All the graphs & tables are rendered (javascript) before the page loader is hidden. They could be moved to be loaded asynchronously. I'll open a new issue for this.

---

I sort of feel like a grumpy old man yelling over here, but you're right, there are different ways of doing things, and they have different advantages and disadvantages. I haven't really considered the option of front side rendering since Plan has used server side rendering from the start.

At the time when the decision was made I had no idea of such things as frontend rendering. Now that I know about it I feel that it would take too much effort to move to such a model when it is easier (for me) to change parts of the server-side rendering instead.

[TomLewis]

haha I know the feeling, I feel exactly the same about JavaScript being this "one use for everything" tool it seems to be nowadays and its 100 new library's appearing every day! But once you start doing some super basic playing with JavaScript you can see how simple things can be, no need for processing data with a server/dynamic scripting language etc.

Tell you what, once this JSON files are much much smaller for each player/server I will use them to populate a player page on my website using just javascript or jquery and show you.

The only issue I can think of at present, is the JSON showing extra information for players I don't want public, at present I have my analytics locked down to just me. is this being too picky to ask for a blacklist for thigns we don't want exported to the JSON haha

[AuroraLS3]

I think a blacklist goes a bit out of scope, since this JSON is not used by Plan. If the json had only single level entries it would be easy, but since it can contain things that go a few levels deep it would need some tree traversal, which is of course doable, but a bit out of scope.

[TomLewis]

Bump. Json file is 531mb, Not really usable in its current state :(

[AuroraLS3]

This one is waiting for interface redesign (#1021) before it is started.

[TomLewis]

its alright, its not like im going to pull my finger out and start development yet hahaha Look at the date difference between ticket replys since I last thought about this, no rush at all. I just pulled the JSON to have a nose and was like 500mb!

[TomLewis]

Excited noises