Stresstest ASL Signbank

[Woseseltops]

Judging from the interest in Instagram, there might be a lot of interest for the ASL Signbank when it launches publically. We better simulate what happens before it actually happens.

[Woseseltops]

I've been looking into this. I've written a simple script that works with Selenium (browser automation), but as noted on many places on the internet this is not optimal because for each test a whole browser has to be opened and closed. As an alternative, I'm looking into Apache JMeter.

[Woseseltops]

Okay I have been working with JMeter for many many hours now, and I've discovered that (1) it is what we need, (2) JMeter is not easy to use, and (3) logging into Django applications from code is HARD: you need the right cookie and the right CRSF token at the right times in the right header, etc.

But I've succeeded! I now have a so called test plan that I can make x virtual users execute spread out within a time frame of y minutes (to prevent that all users do the same action at the exact same time):

• Go to the home page
• Go to the login page
• Log in
• Search for 'lac'
• View the detail view for 'LACHEN-F'

All stylesheets and pictures etc are also downloaded for each page... except, for some reason, the actual video.

Something I've noticed so far is that Signbank becomes slower exponentially if you add more users. These are the average page load times of the Dutch Signbank, for n users doing the actions above spread out over 10 seconds. At 7 simultaneous users I'm also getting timeouts, so I've stopped there.

Curious for your opinions on this experiment, @susanodd @ocrasborn @vanlummelhuizen

[Woseseltops]

Maybe it's also good to write down the technical details of how I got this working. I'll write it down like it's all simple and straightforward, but it actually took me many hours to figure this out.

What you need to log into a Django application:

1. Make a GET request to the page with the login form.
2. Parse the response, and take the CSRF token from the header.
3. Make a POST request to the login url, and include (1) the CSRF token as a parameter and (2) in the header as if it was coming from a cookie. I'm doing the same thing to a redirect URL I'm getting as a response, but I'm not 100% sure if this is necessary.
4. Parse the response, and take a new (!!!) session ID from the header.
5. For all upcoming requests, include the session ID in the header as if it was coming from a cookie.

For all of these requests, it's also important that the 'referer' variable in the header is referring to the previous page.

How to achieve all of this in JMeter:

• The basic workflow for our usecase is to create a 'thread group' ('multi-user session' would be a better name), and below that add 'HTTP Request' samplers for each page we want to visit.
• To set headers, add a 'HTTP Header Manager' below each 'HTTP Request'.
• To give cookie information in the header, just add a new header variable named 'Cookie' and set as a value 'cookie_name_a=cookie_value_a; cookie_name_b=cookie_value_b'. Ignore the Cookie Manager, it did nothing for me.
• To extract information from response headers, add a 'Regular Expression Extractor' to an 'HTTP response request'. You can specify the variable names, which can later be accessed in the 'HTTP Header Manager' via ${varname} .
• The result can be summarized by adding a 'Summary report listener' to the 'thread group'

[Woseseltops]

I did a not-logged in test for ASL Signbank:

• Go to home page
• Go to search page
• Search for 'all'
• View public view of gloss 'all-day'

Surprisingly, this shows a really different pattern:

This created two more questions for which I collected more data:

• Will the home, search and public page be affected by much larger groups of users? Answer: yes, there are limits. I also tried 500 users, but then I got mostly timeouts.

• Is searching slow because of the database queries or because of all the images it has to load? Answer: looks like loading images is a large part of the recorded waiting times, but even without images searching is much slower than the other pages.

But the big question of course is: why is the Global Signbank so much slower? I have three hypotheses:

1. There is much more data to iterate over in the Global Signbank, making all queries to the database much slower.
2. We have configured the Global Signbank in such a way that it does not make use of all the hardware it has.
3. There is some kind of DDOS protection on the RU network.

We could test option 1 by using a temporary, very empty database for the global Signbank, and doing another stresstest.

[vanlummelhuizen]

@Woseseltops Do you have a idea as to where to (stress) test an almost empty database? Do you want to do it in a separate instance on the same server of somewhere else?