search

jonniechow / RogueStockBot

Tracks and updates user whenever Rogue Fitness equipment inventory changes
https://roguestockbot.com/
85 stars 27 forks source link

app.get('/stock-updates', (req, res) => { is inefficient #51

Open criccomini opened 4 years ago

criccomini commented 4 years ago

https://roguestockbot.com/stock-updates is very slow right now. It's taking more than 14 seconds to load. Most of the time is spent in /stock-updates. This code is inefficient:

https://github.com/jonniechow/RogueStockBot/blob/master/app.js#L627-L648

There are a few things going on here that are slow:

  1. All of stock-log.txt is read.
  2. Each line is split twice, once on | and once on ,. String splits are slow.
  3. dataFromLog.item_info is being prepended. This is a slow operation on arrays, as every element has to be copied. Linked lists are faster for this.

Most of this work appears to be throw-away. Only the last 100 items appear to be used (the first 100, after unshifting). There is no pagination either.

This code can be improved in a number of ways. A short term fix would be to trim the stock-log.txt file to just the last 100 lines, though. I took a look, and it appears that the file grows indefinitely. As it grows, the site will get slower and slower as each page refresh will parse all lines in the txt file.

criccomini commented 4 years ago

Alternatively, you could cache the static HTML, with a 30s timeout, or something.

criccomini commented 4 years ago

Something like this might be a quick hack:

https://www.npmjs.com/package/read-last-lines

Stick that at the end of handleAllURLs and read the last 100 lines from the file. You can even reverse lines while you're at it, so you don't need to unshift later.