Wekan performance with 100+users ? #2104

Closed GuiGuiSoft closed 5 years ago

Server Setup Information

Operating System: Ubuntu 16.04 LTS
Deployment Method: Snap
Statistics of 6 last months:

Problem

Hello,

As the number of Wekan users is growing, I am experiencing several performance issues (see statistics above: more than 5 seconds for average page generation).

Despite an hourly restart of the Wekan service (service snap.wekan.wekan restart in crontab), Wekan is often unavailable due to Virtual Machine lack of resource (CPU and RAM are "full").
The VM has 4 CPU and 8Gb of RAM, which seems very important regarding the "small" number of users.
As a comparison, with same resources, a GitLab + Mattermost server is fully functional and responsive for more than 1 000 users.

Do you have any advice to improve the situation?
Are there Wekan servers with more than 200 users ? Do you have the same performance issue ?

Regards.

Hi ! Just for comparaison, we have a docker installation (which for me, save some ressources) with 75+ users and the machine hosting it has 8gb of RAM and 8 CPU. The RAM free is 1gb. We don't experience any interruption of service

Do more than 2 cores help? With our snap deploiment I can't see the benefit of more cores.

we have a docker installation (which for me, save some ressources) with 75+ users and the machine hosting it has 8gb of RAM and 8 CPU. The RAM free is 1gb. We don't experience any interruption of service

How do you plan to scale to 100, 200 or 500 users ?
You tried Snap before Docker and Docker is less resource consuming ?

Do more than 2 cores help? With our snap deploiment I can't see the benefit of more cores.

How many cores do you have ? For how many users ?
If I have to add 2 cores every 25 users, it will be complicated with 500 users 😲

It's not requirement to change from Snap to Docker, if you can wait. Snap still uses Meteor 1.6, that is slower, because I did not get node connecting to MongoDB correctly. There is fix that one Wekan contributor is working on, so that Snap will have faster Meteor 1.8 and MongoDB 4.x. Also, using MongoDB 4.x will make possible to change MongoDB queries to new aggregate queries using features new MongoDB 4.x provides, so this will make Wekan faster. Adding these changes is currently in progress by one Wekan contributor.

If you are in a hurry, for Docker, there is available newer Meteor 1.8 version of Wekan with using MongoDB 4.x, by using image:

image: quay.io/wekan/wekan:meteor-1.8

This meteor 1.8 version should also work with AWS ECS container service and new AWS MongoDB 3.6 compatible scalable database. So you could add ECS cluster to use MongoDB.

How do you plan to scale to 100, 200 or 500 users ? You tried Snap before Docker and Docker is less resource consuming ?

Our Wekan is going to migrate to a new server (which is just a wekan sandbox right now) with 32gb of ram so ressources will not be an issue. I didn't tried wekan with snap so I don't know if wekan with docker is less consuming, I just assumed with my personnal experience between the VM deployment method and the Docker solution. You can save some ressources of your computer without the OS emulation. Nothing to do with wekan

@TiibCD

Resource usage difference is mostly visible only on Windows when using Docker vs Linux Subsystem for Windows. Wekan works best on Linux and there I presume it's hard to notice any resource usage difference of source vs Docker.

Hello,

Thank you for your answers. I understand that:

There is no major performance difference between Docker and Snap.
Docker already use Meteor 1.8 and Snap doesn't use it yet.
Meteor 1.8 is more efficient than the 1.6 version
(I did'nt find any benchmark or article about 1.8 performance improvement. Do you have any source about this subject ??)
You plan to use Meteor 1.8 in Snap package before the end of February
Is there any way to help ? ( @Akuket ?)

Regards.

@Akuket has done only LDAP changes, so this issue not for him. He is mostly busy with his work, most likely I will add his login improvements back.

Some info about Meteor 1.8 is here. For newer modern browsers the bundle of javascript/css/html is much smaller. https://blog.meteor.com/meteor-1-8-erases-the-debts-of-1-7-77af4c931fe3

Info about aggregate queries is here, they have not been added to Wekan yet, but using them will make Wekan faster: https://docs.mongodb.com/manual/release-notes/4.0/

There is no benchmarks yet specifically for Wekan yet. It's experience of one Wekan contributor that has upgraded other Meteor apps to Meteor 1.8.

There is ongoing issue about Meteor performance here. There has been some fixes to Node.js with Node 8.12, but getting fibers working correctly and making more performance improvements for meteor is still on-going process that is being debugged and fixed in newer meteor versions: https://github.com/meteor/meteor/issues/9796

Some more info about various performance improvement options are here: https://github.com/wekan/wekan/wiki/Debugging

In general, Wekan has not been optimized fully yet. Rocket.Chat has same Meteor stack as Wekan does, and Rocket.Chat does not have problems with large amount of users.

For Wekan Snap Meteor 1.8 upgrades, required fix has already made by one Wekan contributor to other Meteor projects.

The way to help Wekan in general is to:

help with adding feature requests and bug reports
adding pull requests to fix any issues, looking at wiki developer docs. Many fixes have been done by beginners.
using my commercial support https://wekan.team for features and bugfixes I plan to do for many years, so I can pay my bills
adding bounties to issues as described at https://wekan.team
help with documenting Wekan https://github.com/wekan/wekan/wiki by directly editing wiki text, and adding attachment images and animgifs as PRs to https://github.com/wekan/wekan.github.io

With more contributors, Wekan would have faster progress. Wekan is community effort.

Hello,

I have heard (read) that NodeJS wasn't very able to use multiple core. So even if I had more and more CPU, performance would not improve.
It seems interesting to run Docker containers in parallel to load-balance the user requests. I will try that and let you know the result.

Regards.

@bentiss

Currently Wekan loads too much html/css/js/data at first time. For example, translations for all languages are loaded, and when importing big boards from Trello and using them in Wekan, Wekan gets very slow, because there is too much data in browserside minimongo. Would it be possible to load only necessary data, for example only in-use translations, and only load data of those cards that are visible, with some infinite load/scroll code? And also, if currently Archived items are not visible, they also would not be loaded to browser. https://atmospherejs.com/?q=infinite

Also, currently importing Trello/Wekan board happens with copy-paste to browser form, at foreground. It would be better to do imports etc bigger things in background for example with https://atmospherejs.com/msavin/sjobs

and just upload JSON file first to MongoDB.

Related: Infinite scrolling at https://github.com/rcdexta/react-trello

@xet7

the infinite scrolling is a nice idea, and I can confirm this reduces the loading time of a big board (thus the implementation in #2144).

Regarding loading the internationalization, are you sure this is what takes time? I am used to good machines with big amount of memory, but still, the tests showed here that the processing of the hidden cards is taking most of the time.

Regarding the concurrent accesses to the database with several users, I ended up having 3 replicates pods on the openshift cluster where the application is hosted. This might not scale up to 200 or more users (we are more 10-20), but this greatly reduces the load between the pods.

@bentiss

No, I'm not sure about internationalization, I have not done profiling.

It would be nice if processing of hidden cards could be prevented.

You could add your openshift changes to https://github.com/wekan/wekan/tree/edge/openshift

You could add your openshift changes to https://github.com/wekan/wekan/tree/edge/openshift

Well, I just used the UI and clicked the up arrow twice to have 3 pods :)

@xet7 is there a recommended set up for hundreds of users?

@xet7 is there a recommended set up for hundreds of users?

just found the large-scale install wiki (https://github.com/wekan/wekan/wiki/AWS)

Related https://github.com/wekan/wekan-mongodb/issues/2#issuecomment-378343587

Hello,

I changed the deployment method from Snap to Docker and I scaled the number of application container to 4.
The results are good : improvement measured via Matomo is about 75% (average generation time decreased from ~10s to ~2.5s with the same number of users).

For those who are interested, here is the docker-compose.yml I use:

version: '3'

services:

  wekandb:
    image: mongo:3.2.21
    command: mongod --smallfiles --oplogSize 128
    volumes:
      - wekan-mongodb_wekan-db:/data/db
      - wekan-mongodb_wekan-db-dump:/dump

  wekan:
    image: quay.io/wekan/wekan:v2.17
    ports:
      - 80:8080
    deploy:
      replicas: 4

    environment:
      - MONGO_URL=mongodb://wekandb:27017/wekan
[ ... some variables related to my needs ... ]
    depends_on:
      - wekandb

volumes:
  wekan-mongodb_wekan-db:
    external: true
  wekan-mongodb_wekan-db-dump:
    external: true

I start and stop the Wekan stack like that:

sudo docker swarm init #init the docker "load balancer" and the ability to replicate containers (it has to executed only one time)
sudo docker stack deploy --compose-file docker-compose.yml wekan #may take few seconds
sudo docker stack rm wekan #stop all containers

It create 4 wekan-app container communicating with 1 wekan-db container. You don't have to use the docker-compose command anymore.

In my case, my conclusions:

NodeJS does not know how to use more than 1 CPU. You can add 30 (virtual) CPU to your VM, it won't change anything if you don't scale Node process (with docker or any Node scalling system like PM2).
With 4 node processes (each around 50% CPU), Wekan is finally usable for end users
Unlike Node, Mongodb is able to use multiples (virtual) CPU without crashing.
With a topcommand, mongodb can reach 300 or 400% CPU without slowing the application.

What next?

I didn't try to scale the wekandb container (I assumed it could lead to problems like concurrent access problems to the database), but it could be the next bottleneck of this solution.
I have to find out why I don't see logs in syslog (will do an other issue for that :) )

@GuiGuiSoft

If you are running at AWS, one option is to use AWS DocumentDB: https://aws.amazon.com/documentdb/

Because AWS DocumentDB is compatible with MongoDB 3.6, it requires using Wekan Meteor 1.8 version, that is at this Docket Image tag:

image: quay.io/wekan/wekan:meteor-1.8

But I have not tested yet how DocumentDB works.

For Azure, there is Azure CosmosDB: https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction

I did not check what MongoDB version Cosmos supports.

Hello, unfortunately I must store my data on my own server (Security policy...) :)

@GuiGuiSoft

Well, in that case, if MongoDB is bottleneck, you could add more CPU to MongoDB VM.

@GuiGuiSoft

But if Wekan is fast enough, I think no more is necessary.

if you scale the mongodb container like @GuiGuiSoft mentioned and have it as a replica set, do your mongodb and mongo_oplog environment variables become: MONGO_URL=mongodb://<user>:<password>@db01:27017,db02:27017,db03:27017/wekan?replicaSet=rs0 MONGO_OPLOG_URL=mongodb://<user>:<password>@db01:27017,db02:27017,db03:27017/local?authSource=admin&replicaSet=rs0

Or do you have to add more options (i.e. read and write concern, timeouts etc.)?

@rjevnikar @GuiGuiSoft @bentiss @TiibCD @GavinLilly

Newest Wekan has 10x lower resource usage, so it's much faster, but is it fast enough for you? Please test.

https://github.com/wekan/wekan/commit/3c49e2d0edec19eff4f87b0fcc127f924af193fc#commitcomment-32537953

Hello @xet7 I can confirm that v2.37 is much better !!!

It is a great step forward because, even with 4 wekan containers replicas, the application was becoming unstable after 2 or 3 days.

The newest build seems to be better for us. The processes still restart once or twice per days but the user interface is very usable. We're still planning on using Kadira to analyse our usage but haven't got round to it yet.

wekan / wekan

Wekan performance with 100+users ? #2104

Server Setup Information

Problem