search

extiverse / bazaar

The extension marketplace for your Flarum forum.
https://discuss.flarum.org/d/5151
MIT License
59 stars 14 forks source link

Bazaar hangs whole forum #116

Closed jsardev closed 6 years ago

jsardev commented 6 years ago

Steps to reproduce

  1. Install Bazaar
  2. Open the Bazaar admin page

Expected behaviour

The page actually loads.

Actual behaviour

The page loads and a modal with "processing" label shows up. The whole forum just hangs in this moment (nobody can access it).

Configuration

Operating system: Debian 9 x64

Version of Bazaar: 0.2.4 Version of Flarum: beta.7 Other extensions: None

Logs: 524: A timeout occurred

Nginx: 2018/05/15 16:15:07 [error] 5292#5292: *258 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xx.xx.xx.xx, server: ***, request: "GET /api/bazaar/extensions?filter%5Bsearch%5D= HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "***", referrer: "https://***/admin"

luceos commented 6 years ago

This is pretty normal. We bulk load too much information with Bazaar in the current version. The version we patched up for beta 8 has many improvements including several fixes for this and background processing of tasks.

jsardev commented 6 years ago

Why would loading a simple listing take ~2 minutes? (2min is my timeout). Aren't you using any search engine for this?

luceos commented 6 years ago

@sarneeh the current bazaar version is very rudimentary, we focussed on solving the primary hurdles first. These include;

Understand that while we wrote the first versions of Bazaar there were hardly 100 extensions.

For the upcoming version we will be patching retrieval of the extensions list to become paginated. We're also moving package actions to cronjob processing. And much more.

Why would loading a simple listing take ~2 minutes? (2min is my timeout).

Because we load the json of over 400 extensions into memory and enrich them with the information the Flarum installation has.

Aren't you using any search engine for this?

No, we used to have a cache in front of our API server, but with the loss of our Ops on the team had to withdraw that layer. We will be improving global performance soon by moving the application to a more scalable solution once we have the funds for it.

jsardev commented 6 years ago

Because we load the json of over 400 extensions into memory and enrich them with the information the Flarum installation has.

So at least it could be done asynchronous. If you're doing some heavy-lifting, async should be the first thing to think of.

No, we used to have a cache in front of our API server, but with the loss of our Ops on the team had to withdraw that layer. We will be improving global performance soon by moving the application to a more scalable solution once we have the funds for it.

You don't need a Ops in the team to handle this kind of things. You also don't need funds for it as you're an open-source project. Just take a solution like https://www.algolia.com (free for oss) and make it fly.

But if you're talking about funds in terms of funding the time you guys work on it - then yeah, I understand that this is something that doesn't take 1 hour of work to be done and no one wants to work for free all the time 😄

I don't want to blame anyone here, I understand that many of these tools are fresh and rudimentary. You're all doing a great job around the Flarum community. I just wanted to understand the decisions behind the solutions and maybe point out some practices that could be easily improved.

jsardev commented 6 years ago

Oeps, my bad, seems like Algolia isn't actually free for oss 😭

luceos commented 6 years ago

There are a few factors that remove the ability to easily cache the full extensions list:

Don't get me wrong, we don't need an ops necessarily. We do need time or an (dev)ops to improve the situation.