Improve performance with queuing and caching

[cydrobolt]

Polr 2.x suffers from some performance issues under high load. We can significantly improve performance and reduce bottlenecking by implementing a queue (e.g redis, rabbitmq) for analytics data collection on click, and by caching requests in redis. The performance hit is especially visible when advanced analytics are used.

The cache would reduce the amount of database transactions we make and reduce latency, especially during peak periods. We would need to implement an event that clears links out of the cache when they're updated, and a queue task that inserts the analytics data into the database once the user visits the link. Queueing tasks is especially important when the advanced analytics setting is enabled.

100 connections on single-threaded development server:

Running 10s test @ http://localhost:8000/AqH6g
  8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.03s   606.95ms   1.99s    56.82%
    Req/Sec     8.58      5.65    30.00     77.58%
  188 requests in 10.03s, 172.10KB read
  Socket errors: connect 0, read 188, write 0, timeout 144
Requests/sec:     18.75
Transfer/sec:     17.16KB

10 connections on single-threaded development server:

Running 10s test @ http://localhost:8000/AqH6g
  4 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   379.11ms   52.47ms 484.66ms   85.44%
    Req/Sec     6.08      2.94    10.00     37.58%
  206 requests in 10.01s, 188.54KB read
  Socket errors: connect 0, read 206, write 0, timeout 0
Requests/sec:     20.57
Transfer/sec:     18.83KB

We can vastly improve the performance during high loads by removing the bottlenecks. I believe #255 is relevant, although it needs to be updated.

[columbian-chris]

For Lumen 5.1.6 (what appears to be currently installed): https://lumen.laravel.com/docs/5.1/cache

If the version of Lumen was upgraded: https://lumen.laravel.com/docs/master/cache