Combine json files on server and optimize performance for chart loading

[elcojacobs]

All json files for a beer are downloaded separately and combined in javascript. Combining them on the server might be faster, because it does not require lots of ajax requests.

Brian got started in the feature branch optimize-beer-data-loading.

I fixed some errors in the resulting json and the branch now works correctly. (https://github.com/elcojacobs/brewpi-www/commit/11b36283a87709d101473d7627b0c65933bd8c83).

I am now profiling the file with xdebug and webgrind. json_decode and json_encode are the heaviest functions. The files that are loaded are already json, so I think we can avoid decoding and encoding.

[elcojacobs]

I replaced the json_decode and json_encode with string parsing: https://github.com/BrewPi/brewpi-www/commit/df03c83a2cc587eb1504fa4edcf1126554dfa91f

The time is down from 614 to 70ms on wamp:

[elcojacobs]

I now also removed the datatable and the dependencies that came with it (google jsapi). The data is now parsed to Dygraph native array format. More testing is needed, and I would like to get rid of the use of eval()

https://github.com/BrewPi/brewpi-www/commit/deaef6885f999e9fcff28cf920cb3749363699f0

[elcojacobs]

The annotations should be passed to the chart separately, I am working on that part.

[elcojacobs]

Annotations now work again :smiley: Please take a look at the code, I am sure it can be improved. 63be43b849a7465fb35acfdd6b2917a750d8146a

[wfowlks]

I was running pagespeed because I noticed that when a beer was going for several days, there was great latency when loading the graph and on the page. The Ajax is not asynchronous so it prevents me from changing the temperature before the graph is loaded which lately has been 6+ seconds just for get_beer_data.php, I will take a stab at this and see if I can help at all.

Running Google's PageSpeed plugin for chrome, Google suggested minifying the CSS files as well as the javascript files. Might it be possible to minify for production but leave the full css for development?

[jhferris]

wfowlks,

minifying the js and css are highly unlikely to appreciably improve your page load times (from 6 seconds, anyway)

I suspect the core problem is that the get_beer_data.php bit takes a while on the server because of the way it loads/transforms the data.

[wfowlks]

jhferris,

Minification gave me a 1/2 second improvement on load times on my desktop and 1 second for mobile. But I do concur that the main latency is the data transformation.

Do you think that utilizing SQLlite instead of having a json log file for each day might be an improvement. THe main Pagespeed suggestion was to utilize the browser cache.

[elcojacobs]

For next gen, we think influxdb might be a good database to store the timeseries data. The chart could then query for a subset of the data and get more datapoints when the user zooms in.

After getting the data in influxdb, grafana might be a nice configurable dashboard to display charts.

[wfowlks]

I spent some time last night trying to install influxdb on my pi, however it was pretty painful and I was unsuccessful. I tried using the prebuilt binaries, as well as building from scratch. If you are successful can you let me know what you did, because I'd like to try it out and see if there are any performance improvements.