Pact-stub-server Multi-threaded

[Ajay2020813]

Hi Team

I was trying to run the pact stub server with multiple instances of the client but i am getting errors related to request timeouts.Looks like PactsubServer is not capable of serving the parallel requests.

I am using swift app with multiple simulators to see the behavior.

Is there any setting/param to be enable to make it support multi-thread?

Any support is appreciated. This is urgent

Thanks

[uglyog]

Looks like it was CPU bound (it was only running one thread, and that thread/CPU was being overwhelmed). I have pushed a fix for this.

[Ajay2020813]

Thank you so much for the quick fix.

On Mon, Jul 13, 2020 at 5:57 PM Ronald Holshausen notifications@github.com wrote:

Looks like it was CPU bound (it was only running one thread, and that thread/CPU was being overwhelmed). I have pushed a fix for this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-657904869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T65AKJEEP2YKFS4P7E3R3OUPXANCNFSM4OXKTV7A .

[uglyog]

0.4.2 has been released with the fix

[Ajay2020813]

Hi Ronald

I am still seeing the pact stubby server is not responding well as it was single client/app simulator. Definitely there is improvement, but still lot of time out comes.

Pact stubby server responds with more errors if I increase more than 2 clients.

Is there any way there could be more optimization on the server to handle multiple requests better or if require, Can we allocate more resources to this process on Mac machine?

Appreciate your insight

Thanks AJayGupta

On Mon, Jul 13, 2020 at 6:30 PM Ronald Holshausen notifications@github.com wrote:

0.4.2 has been released with the fix

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-657913590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T67QU7SSPJHGCDVI6N3R3OYJ3ANCNFSM4OXKTV7A .

[uglyog]

How are you running the test, and what is the machine CPU and memory usage while it is running?

I ran a test with 5000 requests per second. This was on the loopback adaptor, so no network was involved.

$ echo 'GET http://127.0.0.1:8080/activities' | vegeta attack -duration=10s -rate=5000 | tee results.bin | vegeta report
Requests      [total, rate, throughput]         50000, 5000.11, 5000.00
Duration      [total, attack, wait]             10s, 10s, 226.018µs
Latencies     [min, mean, 50, 90, 95, 99, max]  119.339µs, 385.528µs, 218.218µs, 274.204µs, 311.223µs, 4.859ms, 39.723ms
Bytes In      [total, mean]                     8200000, 164.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:50000

By default, it will use 1 thread per CPU core plus one, so on my laptop it would use 13 threads. Your machine will be different.

[Ajay2020813]

Ronald Thanks for the response!

I would like to show you live the behavior we are observing.

When could you spare some time ? I will set up a meetup.

It will be my pleasure to meet with you and discuss this further.

Thanks AjayGupta

On Tue, Jul 14, 2020 at 9:31 PM Ronald Holshausen notifications@github.com wrote:

How are you running the test, and what is the machine CPU and memory usage while it is running?

I ran a test with 5000 requests per second. This was on the loopback adaptor, so no network was involved.

$ echo 'GET http://127.0.0.1:8080/activities' | vegeta attack -duration=10s -rate=5000 | tee results.bin | vegeta report

Requests [total, rate, throughput] 50000, 5000.11, 5000.00

Duration [total, attack, wait] 10s, 10s, 226.018µs

Latencies [min, mean, 50, 90, 95, 99, max] 119.339µs, 385.528µs, 218.218µs, 274.204µs, 311.223µs, 4.859ms, 39.723ms

Bytes In [total, mean] 8200000, 164.00

Bytes Out [total, mean] 0, 0.00

Success [ratio] 100.00%

Status Codes [code:count] 200:50000

By default, it will use 1 thread per CPU core plus one, so on my laptop it would use 13 threads. Your machine will be different.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-658539196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T62JNNEK4RN5ADZWIU3R3UWLVANCNFSM4OXKTV7A .

[Ajay2020813]

Hi Ronald

One more thing to add is our pact-stub-server is loaded with pact files of around 10 MB, it’s basically around 1000 interactions.

But we are also running on the same machine where client app is running, so no network involves.

Thanks AjayGupta

On Tue, Jul 14, 2020 at 7:11 PM AJay GUpta aj.kumar.gupta@gmail.com wrote:

Hi Ronald

I am still seeing the pact stubby server is not responding well as it was single client/app simulator. Definitely there is improvement, but still lot of time out comes.

Pact stubby server responds with more errors if I increase more than 2 clients.

Is there any way there could be more optimization on the server to handle multiple requests better or if require, Can we allocate more resources to this process on Mac machine?

Appreciate your insight

Thanks AJayGupta

On Mon, Jul 13, 2020 at 6:30 PM Ronald Holshausen < notifications@github.com> wrote:

0.4.2 has been released with the fix

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-657913590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T67QU7SSPJHGCDVI6N3R3OYJ3ANCNFSM4OXKTV7A .

[Ajay2020813]

Hi Ronald

I did quick 2 stress test and here are the results

echo 'GET http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781' | vegeta attack -duration=1s -rate=100 | tee results.bin | vegeta report

Requests [total, rate, throughput] 100, 101.02, 3.91

Duration [total, attack, wait] 25.597s, 989.892ms, 24.607s

Latencies [min, mean, 50, 90, 95, 99, max] 888.178ms, 13.91s, 14.972s, 23.209s, 24.034s, 24.853s, 24.957s

Bytes In [total, mean] 37900, 379.00

Bytes Out [total, mean] 0, 0.00

Success [ratio] 100.00%

Status Codes [code:count] 200:100

Error Set:

echo 'GET http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781' | vegeta attack -duration=1s -rate=500 | tee results.bin | vegeta report

Requests [total, rate, throughput] 500, 501.27, 2.61

Duration [total, attack, wait] 30.992s, 997.47ms, 29.994s

Latencies [min, mean, 50, 90, 95, 99, max] 58.383µs, 12.311s, 455.881ms, 30s, 30s, 30.001s, 30.003s

Bytes In [total, mean] 30699, 61.40

Bytes Out [total, mean] 0, 0.00

Success [ratio] 16.20%

Status Codes [code:count] 0:419 200:81

Error Set:

Get "http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781": dial tcp 0.0.0.0:0->127.0.0.1:9992: socket: too many open files

Get "http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

AFTER ulimit -Sn 20000

echo 'GET http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781' | vegeta attack -duration=1s -rate=500 | tee results.bin | vegeta report

Requests [total, rate, throughput] 500, 501.13, 2.71

Duration [total, attack, wait] 30.998s, 997.735ms, 30s

Latencies [min, mean, 50, 90, 95, 99, max] 2.058s, 27.326s, 30s, 30s, 30s, 30.001s, 30.005s

Bytes In [total, mean] 31836, 63.67

Bytes Out [total, mean] 0, 0.00

Success [ratio] 16.80%

Status Codes [code:count] 0:416 200:84

Error Set:

Get "http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781": EOF

Get "http://127.0.0.1:9992/orders/nonmerch/brand/10/market/US/store/04781": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

HERE IS MY LAPTOP CONFIGS

Ajays-MacBook-Pro:Sell ajay$ ulimit -a -S

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

file size (blocks, -f) unlimited

max locked memory (kbytes, -l) unlimited

max memory size (kbytes, -m) unlimited

open files (-n) 20000

pipe size (512 bytes, -p) 1

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 1418

virtual memory (kbytes, -v) unlimited

Ajays-MacBook-Pro:Sell ajay$ ulimit -a -H

core file size (blocks, -c) unlimited

data seg size (kbytes, -d) unlimited

file size (blocks, -f) unlimited

max locked memory (kbytes, -l) unlimited

max memory size (kbytes, -m) unlimited

open files (-n) unlimited

pipe size (512 bytes, -p) 1

stack size (kbytes, -s) 65532

cpu time (seconds, -t) unlimited

max user processes (-u) 2128

virtual memory (kbytes, -v) unlimited

Please provide the guidance, we are blocked here. Any help is appreciated!

Thanks

Ajay

On Wed, Jul 15, 2020 at 4:00 AM AJay GUpta aj.kumar.gupta@gmail.com wrote:

Hi Ronald

One more thing to add is our pact-stub-server is loaded with pact files of around 10 MB, it’s basically around 1000 interactions.

But we are also running on the same machine where client app is running, so no network involves.

Thanks AjayGupta

On Tue, Jul 14, 2020 at 7:11 PM AJay GUpta aj.kumar.gupta@gmail.com wrote:

Hi Ronald

I am still seeing the pact stubby server is not responding well as it was single client/app simulator. Definitely there is improvement, but still lot of time out comes.

Pact stubby server responds with more errors if I increase more than 2 clients.

Is there any way there could be more optimization on the server to handle multiple requests better or if require, Can we allocate more resources to this process on Mac machine?

Appreciate your insight

Thanks AJayGupta

On Mon, Jul 13, 2020 at 6:30 PM Ronald Holshausen < notifications@github.com> wrote:

0.4.2 has been released with the fix

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-657913590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T67QU7SSPJHGCDVI6N3R3OYJ3ANCNFSM4OXKTV7A .

[uglyog]

Unfortunately, I only get to work on this project in my spare time.

Can you explain what you are trying to achieve. The stub server was written to support prototyping new consumers, and not for performance testing. How many requests per second are you making to it, and what is the CPU and memory load of the machine while you are running the test?

[uglyog]

If you have a pact file with 1000 interactions, the stub server will have to match every incoming request against 1000 possible ones. That is a lot of work it has to do.

[uglyog]

dial tcp 0.0.0.0:0->127.0.0.1:9992: socket: too many open files probably means your terminal is not handling the amount of console output. You need to turn off the logging with the stub server if you want to run this type of load.

[uglyog]

I've done some performance enhancements. HTTP keep alive was disabled, so that might have been causing the timeouts. Try version 0.4.3 and see how it goes.

[Ajay2020813]

Thanks Ronald!

I will try and let you know!

On Sat, Jul 25, 2020 at 7:53 PM Ronald Holshausen notifications@github.com wrote:

I've done some performance enhancements. HTTP keep alive was disabled, so that might have been causing the timeouts. Try version 0.4.3 and see how it goes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/uglyog/pact-stub-server/issues/36#issuecomment-663928838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQH3T67H2DGH3GYSD76W4XDR5OLDDANCNFSM4OXKTV7A .