More progressive extension loading from flagrow.io

[clarkwinkelmann]

With the work started in #83 it should be possible to move away from the arbitrary 9999 page size and load packages in smaller chunks.

Here's the idea:

Instead of downloading all the extensions from flagrow.io into the cache before answering the API call to show the extensions in the admin panel, we could only load as many extensions as we display in a single page in Bazaar.

Then AJAX calls (or ideally queued Jobs but I doubt most users would support those) could be used to continue updating the cache in the background. If the user then filters or scrolls the list of extensions (most of) the remaining extensions should have been loaded in the cache and will be shown.

With this system we can probably achieve faster loading time the first time the user visits Bazaar after the cache expired, and this also allows to download the list of extensions in chunks instead of doing a massive request like we do right now.

Instead of clearing the cache when it expires, the same AJAX system could be used to update the cache progressively. Each extension data would be stored along with a timestamp. When the updated data is fetched this timestamp is updated. If an extension survived one or two updates without its timestamp being updated, it could be deleted because it means it's no longer listed by the flagrow.io API.

Another idea I remember discussing with @luceos (but not sure if we wrote it somewhere) was to implement pointer-based updates on flagrow.io so that Bazaar can asks for "what has changed" since the last request instead of downloading every single extension detail again. This would probably be a lot more resource-intensive on flagrow.io though.

[luceos]

pagination was added, might need a bit more fine tuning, but let's close this for now