Load cached sim data for presets

[ezio-melotti]

Since the results for presets are known, it would be better to cache them instead of recalculating them every time.

This can be done by having the sim-data files on a fixed location (e.g. simoc.space/sim-data/xxx.json) and downloading them from there when a user selects one of the presets and clicks on Launch simulation.

We could also use GitHub actions to download the files and include them in the container before building the frontend, so that they end up being hosted on and server through GCP (including them directly in the repo is technically easier, but it will end up bloating the repo). Compressing the files will also make the download faster.

@kstaats I attached the 4 sim data files, one for each of the 4 presets. I had to change the extension from .json to .txt because apparently GitHub doesn't like JSON files, but if you can change it back to .json it would be better. I think we should have a dir (such as sim-data/ or simdata/) for these files, and in future another dir (such as presets/) for the presets.

simoc-simdata-4-human-garden-preset.txt simoc-simdata-4-human-preset.txt simoc-simdata-1-human-radish-preset.txt simoc-simdata-1-human-preset.txt

[ezio-melotti]

The files have been uploaded at https://simoc.space/download/simdata/, however I can't access them from the frontend. The issue seems related to CORS, and apparently to fix it the server header Access-Control-Allow-Origin: * must be added. @kstaats, can you look into it?

[ezio-melotti]

I had to update the simdata files after the changes introduced in e93aa45.

@kstaats can you replace the files you uploaded on simoc.space with these ones (and change the extension from .txt to .json)?

simoc-simdata-4-human-garden-preset.txt simoc-simdata-4-human-preset.txt simoc-simdata-1-human-radish-preset.txt simoc-simdata-1-human-preset.txt

[ezio-melotti]

This is a rough summary of the tests and findings I did:

• putting the simdata on simoc.space and download them directly -> CORS issues
• putting the simdata on ngs.simoc.space -> solves the CORS issues but:
  • downloading the files directly from ngs -> causes CORS issues between localhost and nginx locally
  • bundling the files with the frontend -> lot of data to load on startup (15+ MB)
  • lazy loading through webpack -> works but
  • dynamic loading of modules requires literal expressions at compile time in require() -> done

Since the simdata files are big, they need to be fetched instead of having them in the repo:

• [x] locally by wgetting the files in src/assets/simdata/ from run.sh (run when the container is started) -> done
• [x] in production by wgetting the files through github actions before running npm run build -> done (see https://github.com/overthesun/simoc/pull/127)

Some new issues surfaced from this:

• [x] The simdata files didn't include the additional data sent from the server before sending the steps. This broke the atmospheric level panel -> fixed in #71
• [x] When loading a simdata file and quitting from the dashboard, the FE still tells the BE to kill game and gets back an error:
  • [x] the FE shouldn't tell anything to the BE in this case; -> done
  • [x] the BE shouldn't give an error when the gameid is missing; -> not done, but shouldn't be an issue anymore
  • [x] investigate whether the gameid should be included in downloaded simdata files; -> now hidden if missing
  • [x] possibly hide the mission/game id from the mission info panel ifit's missing; -> done
• [x] A yet-to-be-investigated issue about a duplicate index while loading simdata files; -> haven't seen this again
• [x] Loading icon breaks while hitting the "Launching simulation" button with a preset apparently due to the synchronous/blocking JSON import code:
  • related to VueJS Async Update Queue not being updated
  • fixed with a workaround
  • works while requesting a new game from the server due to an async axios call

Other solutions investigated:

• instead of storing simdata files on simoc.space, they could be stored on GitHub:
  • in an issue/PR or maybe somewhere else (wiki? releases?);
  • in the repo using git-lfs;
  • this should let us store big files without bloating the repo;
  • possibly return to this later on;

[ezio-melotti]

71 has been merged, but upon further testing on the production server it appears that all the presets are bundled and sent to the frontend, instead of being sent on-demand when the preset is selected. This increases the initial loading time, and should be investigated and fixed.

[ezio-melotti]

73 fixes the problem reported in the previous message.

---

I also spent some time trying to figure out if the webpack bundle could be further optimized, without much success. I found and installed https://github.com/webpack-contrib/webpack-bundle-analyzer following the instructions in the README. Using it as a plugin inside Vue doesn't seem to work. Using it from the command line doesn't seem to work either. I tried to run webpack --profile --json > stats.json but it doesn't recognize the command. I installed webpack and it run the command, but it gave the error Entry module not found: Error: Can't resolve './src' in '/frontend'\nresolve './src' in '/frontend'\n using description file: /frontend/package.json (relative path: .)\n. Even though we have a /src dir, it apparently expects an index.js in it, so I added it (see https://stackoverflow.com/a/49772864). The command succeeded but then webpack-bundle-analyzer bundle/output/path/stats.json failed, because it can't find the command. Apparently npx webpack-bundle-analyzer dist/stats.json works (see https://stackoverflow.com/a/50060552), but in the meanwhile I also found https://github.com/statoscope/statoscope/ and uploaded the stats file at https://statoscope.tech/#&hash=9068ee5518a5819783bf, but apparently it just created the stats for the empty index.js I added. Then I found and installed https://github.com/mrbbot/vue-cli-plugin-webpack-bundle-analyzer which seems specific to Vue, and even though I followed the README to enable the plugin, it doesn't say what to do to see the output.

---

Another option is to enable gzip compression, and to do that it seems that:

1. an npm package needs to be installed on the frontend (I've already seen a couple but I haven't tried them);
2. gzip compression must be enabled on the nginx config;

However I think in production we use traefik instead of nginx. I think it might be worth testing a frontend plugin with nginx, and if it works try to deploy it in production. Traefik might use gzip out of the box, but if not, we can figure out how to enable it.

[ezio-melotti]

This got fixed in:

• 383

• https://github.com/overthesun/simoc/pull/315

See also:

• 359

• https://github.com/overthesun/simoc/pull/297