Artillery + Playwright Performance for scale

[stanyq4]

I have already come across a few similar discussions, but I am curious to know if any significant progress/improvement has been made.

I am trying to run a load test using a Playwright script through the Artillery framework, but I am quite surprised by how resource-intense those runs are, e.g. In order for me to run a test with an arrivalRate of 100+, I need to spin up 200 Fargate containers with 16CPU and 64 GBs of RAM, where the test itself is very simple, just land on the page and fill out a registration form.

I've tried a few optimizations e.g. headless, some chromium flags, but haven't achieved any significant improvement.

I am wondering if anyone already had a success story and can share details of their setup, and for now I am curious about the following:

1. What's the most lightweight browser/browser configuration I can use?
2. Can I run Fargate using Arm architecture?
3. Are you planning to add Lamdda support?

[vikramjb]

@stanyq4

There is support for Lambda. Documentation link provided below

https://www.artillery.io/docs/load-testing-at-scale/aws-lambda

[stanyq4]

Hi @vikramjb

I am specifically interested in the Playwright engine, which is not supported for lambda yet.

[hassy]

Hey @stanyq4 👋 A change was made in Artillery v2.0.4 to use browser contexts for VUs to reduce resource usage. Worth an upgrade if you're not on latest already.

There's a limit on how much optimization can be made in Artillery (or even Playwright) as ultimately you are running a full browser and that's where most of the overhead is. Scaling out horizontally is the answer, and Artillery makes that very easy.

To answer your questions:

1. Only Chrome is available in the Artillery + Playwright integration right now. I'd be curious to see the difference in resource consumption vs Safari, but it's not something we ourselves have the bandwidth to look at right now.
2. We don't support ARM on Fargate yet, but planning to add it.
3. We do, but no ETA on that yet. There's some other work that needs to be done first, like changing from zip-based Lambda functions to ones based on Docker images

For your specific use case - if the test itself is to just fill out and submit a form and not something more complex, why not just use plain HTTP calls to load test the backend?

[stanyq4]

Thank you for the detailed response, @hassy

I am already on version Artillery: 2.0.8, so I am already utilizing the useSeparateBrowserPerVU flag.

In terms of the load test with HTTP calls, that's what we've been doing for years, but as our app grows, even for simple form submission, there is a lot more additional logic added, and we now have up to 15 HTTP calls and one web socket connection as part of the user journey. Instead of constantly guessing all the endpoints and their timings, we wanted to do a real browser test.

Looking forward to Lambda support, I believe this would solve our problem as it does today for HTTP-based load tests.

Thank you

[hassy]

That makes sense @stanyq4, exactly the kind of use case we added Playwright support for :)

Just want to check - are you setting useSeparateBrowserPerVU to false? The default behavior since v2.0.4 is to use browser contexts (the more lightweight option). Setting useSeparateBrowserPerVU to true will actually revert back to the old heavyweight behavior of creating a whole new browser instance for each VU.

[stanyq4]

@hassy I am not setting useSeparateBrowserPerVU at all, so I presume I am using false as a default. :)