global: Add performance/scalability/stress tests

[nharraud]

We need to add a few tests to check for the performance of B2Share. Locust seems to be a simple enough solution.

These tests would only check the REST API.

depends on #1126

Tests to perform:

• [ ] run CLI commands once many records and users have been created.
  • why: It's possible that some commands fail because they try to fetch every record or every user in one request.
  • solution if it fails: optimize functions like list_db_published_records in order to stream results. See this article for a better explanation.

This issue is about testing. A separate issue should be created for each scalability/performance problem once detected.

[emanueldima]

A minimal such test should be performed before upgrading CSC production site.

[nharraud]

Before these tests are done the deployment should be improved, see #1126.

Minimal test:

• multiple simultaneous uploads of small, medium and big files.
• logins and logouts
• record editions, and publications.
• many embargoes finishing at the same time or nearly at the same time.

What else?

[emanueldima]

• many concurrent searches and community, schemas, records retrievals

[llehtine]

Now there is new server up&running with b2share 2.0 evolution for the stress-testing @ https://eudat-b2share-test.csc.fi/ so the demo-site itself don't need to be flooded. Also, the new server will have similar setup as the new production server where we are going to move so it's better to use that one for testing.

[nharraud]

Thanks a lot @llehtine !

Can you provide the approximate config of the server? Just to have an idea of the number of containers and services we can run on it.

[llehtine]

It has 1 vCPU with 4GB RAM / 20GB Amounts can be increased if necessary. Do you think the disk amount would be enough for this kind of purpose?

[nharraud]

Before asking for more I need some more information:

• 1. What are the performance requirements for CSC B2Share? Needed information include: number of users uploading at the same time, number of records, number of communities, number of search requests, expected downtime (the CSC service will not be resilient as it will be down as soon as this single server is stops)... not just now but also in the next two years. It doesn't need to be precise but we need to know approximately what is expected.
• 2. Same question but for the biggest instance which we know will exist in the short/medium term plan.

From the specifications of this server it looks like there will not be many people using the server (which is currently the case). However I am worried that this might not represent B2Share's max requirements (B2Share software, not the CSC service).

We also have to take into account future features. If we later do tasks such as generate data previews, the next server will have to be more powerful. How much more powerful depends on the previous questions.

@thijscobben @emanueldima @SarahBA @llehtine Who can I ask for those information?

[emanueldima]

@nharraud Nobody can tell you, but we can find a probable maximum based on the number of dropbox users and percent of researchers.

[nharraud]

@emanueldima Ok, I'll ask Chris Ariyo and Mark. There should be somebody knowing at the very least how many users should one instance of B2Share support in the future. If there isn't, we must put pressure to have this information, otherwise we can't scale.

Knowing how many users we have today is not enough as it won't prepare us for the V2 release.

[nharraud]

If nobody can give an approximation, the best approach IMHO would be to test B2Share with Zenodo's requirements. This would be more realistic.

[nharraud]

depends on #1126

[thijscobben]

This is in requirements gathering phase. This also has to do with Europeana and setting up test environment for performance testing. This will probably be an ongoing issue. I will try to create smaller, deliverable, tasks out of it.

[nharraud]

@thijscobben I started listing tests, in the issue's description, which might fail once a B2SHARE instance is scaled.