Proposal: ramp-up request rate

[ghost]

In this AWS document: http://aws.amazon.com/articles/1636185810492479, it states: "We recommend that you increase the load at a rate of no more than 50 percent every five minutes. Both step patterns and linear patterns for load generation should work well with Elastic Load Balancing"

I thus propose a ramp-up with an the interface @tsenart has proposed on the project Gitter chat:

vegeta attack -plan="10:20s 100:20s 1000:20s"

[tsenart]

With some thinking to do about the command line interface, I'm in favour of this proposal.

[xakraz]

+1

[ghost]

Hi @tsenart

The ramp-up is becoming increasingly important to us. I wondered if you could give me some pointers, so I could attempt to implement this myself?

[tsenart]

Hello @aidylewis,

I'm sorry, I haven't had time to work on Vegeta recently. I took 2 hours to implement this feature in the plans branch. It's not well tested and probably needs some work with a clearer mind to get all the edge cases and quality sorted, but you can check it out, build it, test it, etc :-)

[tsenart]

Here's an example invocation:

echo "GET http://:6060" | ./vegeta attack -timeout=6s -duration=20s -plan="100@2s 200@4s 400@6s 500@8s 600@10s" | tee result.bin | ./vegeta report

[ghost]

This is fantastic. We appreciate this so much. I'll test it out tomorrow. Thanks.

[ghost]

For anyone else who would like to test this branch:

go get github.com/tsenart/vegeta
cd $GOPATH/src/github.com/tsenart/vegeta
git checkout plans
go install github.com/tsenart/vegeta

[ghost]

Hi @tsenart

• We cannot add individuals plans of the same duration.

-plan="100@20s 200@20s”

> 2016/02/04 09:15:02 bad timing order in plan "100@20s 200@20s"

• How is the ramp-up supposed to work?

Will the below start off at 100 r/s and then after ten seconds ramp-up to 200 r/s?

echo "GET http://whatever" | vegeta attack -duration=30s -plan="100@10s 200@20s" > results.bin

We never make it to 200 r/s.

Requests      [total, rate]            3989, 133.03
Duration      [total, attack, wait]    29.999565631s, 29.985s, 14.565631ms

• Do you think we need a duration flag when we have a plan?

[tsenart]

I think you misunderstood the semantics. In a plan, you define relative points in time where the rate should change. For instance, "100@10s 200@20s" means that at 10s into the attack, the rate will change to 100 and at 20s into the attack, the rate will change to 200.

The duration is relevant due to these semantics. On Thu, 4 Feb 2016 at 10:56, aidylewis notifications@github.com wrote:

Hi @tsenart https://github.com/tsenart

• We cannot add individuals plans of the same duration.

-plan="100@20s 200@20s”

2016/02/04 09:15:02 bad timing order in plan "100@20s 200@20s"

• How is the ramp-up supposed to work?

Will the below start off at 100 r/s and then after ten seconds ramp-up to 200 r/s?

echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=30s -plan="100@10s 200@20s" > results.bin

We never make it to 200 r/s.

Requests [total, rate] 3989, 133.03 Duration [total, attack, wait] 29.999565631s, 29.985s, 14.565631ms

• Do you think we need a duration flag when we have a plan?

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179739572.

[ghost]

Hi @tsenart

Thanks for your response.

If at 10s into the attack the rate change will be 100, what will be the rate from 0 - 10s?

[tsenart]

As it stands, it would be 100, since you didn't specify any previous rate. To do so, you can specify a rate at t=0. e.g. 50@0 100@10s 200@20s

As you know, I did this in two rushed hours. Please give me any feedback you have on the interface, expectations, etc.

[ghost]

The interface is sensible, I just misunderstood it. I'll do some testing and ask @DaveBlooman from News to do some testing/give feedback as they are also interested in the functionality.

[ghost]

In fact, I think we need an interface where we specify the ramp-up over a specific period of time and the constant rate over a specific period of time and remove the duration. But I am not sure how to convert that into command line options.

rampto=200for5m constant=200for15m # which is ugly

Thoughts @daveblooman & @gunisp01?

[tsenart]

I don't see how you can't achieve that with the current interface. On Thu, 4 Feb 2016 at 14:27, aidylewis notifications@github.com wrote:

In fact, I think we need an interface where we specify the ramp-up over a specific period of time and the constant rate over a specific period of time and remove the duration. But I am not sure how to convert that into command line options.

rampto=200for5m constant=200for15m # which is ugly

Thoughts @daveblooman https://github.com/daveblooman & @PoornimaGunisetti?

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179834800.

[gunisp01]

We can make it work with current interface as 50@0 100@10s 200@20s 200@20m

[tsenart]

Did you mean 2000@20m? Because if it's 200@20m then there would be no change from 200@20s. On Thu, 4 Feb 2016 at 14:30, Poornima Gunisetti notifications@github.com wrote:

We can make it work with current interface as 50@0 100@10s 200@20s 200@20m

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179836885.

[gunisp01]

Nope I meant 200@20m only as we need it to ramp up within first few seconds and there after wait for few minutes (constant rate). In case we do not put 200@20s and 200@20m, it will take 20 minutes to ramp up to 200 which is not something we need. Correct me if I am wrong in understanding @aidylewis requirement here!

[tsenart]

You can drop the last one. 50@0 100@10s 200@20s with a large enough attack duration should do the trick. On Thu, 4 Feb 2016 at 14:37, Poornima Gunisetti notifications@github.com wrote:

Nope I meant 200@20m only as we need it to ramp up within first few seconds and there after wait for few minutes (constant rate). In case we do not put 200@20s and 200@20m, it will take 20 minutes to ramp up to 200 which is not something we need. Correct me if I am wrong in understanding @aidylewis https://github.com/aidylewis requirement here!

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179842961.

[gunisp01]

Ignore me, I just realised it is the time at which "ramp up will occur" rather than "ramping up" time.

[ghost]

Yes, the interface is fine. We'll do some testing.

[tsenart]

The attack rate between plan transitions is always constant. So for "50@0 100@10s 200@20s" with an attack duration of 30 min, the rates would be.

0s until 10s: 50 10s until 20s: 100 20s until 30m: 200 On Thu, 4 Feb 2016 at 14:43, Poornima Gunisetti notifications@github.com wrote:

Cool, makes sense. But what if we need constant rate between ramp up? How can we achieve that?

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179846384.

[ghost]

Example:

shell_1

echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=120s -plan="5@0 10@30s 20@60s" > results.bin

shell_2

while true; do cat results.bin | vegeta report ; sleep 2; done 

[ghost]

In the above attack, Vegeta is not managing to reach 10 r/s from 30s to 60s and we never reach 20 r/s

[tsenart]

How are you verifying this? On Thu, 4 Feb 2016 at 15:44, aidylewis notifications@github.com wrote:

In the above attack, Vegeta is not managing to reach 10 r/s from 30s to 60s and we never reach 20 r/s (even though we have given 60 seconds to achieve this throughput).

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179877022.

[ghost]

Using the above bash loop which pipes the results.bin into the Vegeta report.

[ghost]

Although, I can't see it in the code, you probably average the current total rate, when I am thinking it is the current rate.

[tsenart]

No. It's the average rate, not the last one. On Thu, 4 Feb 2016 at 19:37, aidylewis notifications@github.com wrote:

Although, I can't see it in the code, you probably average the current total rate, when I am thinking it is the current rate.

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179990291.

[ghost]

My fault. I will try tcpflow or something tomorrow.

[ghost]

I am using httpry on a OS X network interface at 1 second intervals.

sudo httpry -i en0 -s -t 1

We don't get to 10 r/s between 30 - 60 secs, and don't reach 20 r/s.

echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=120s -plan="5@0 10@30s 20@60s" > results.bin

[tsenart]

I'm not sure that tool is measuring the right thing right. Here's a short Go HTTP server program which should show you how many requests per second it is receiving.

package main

import (
    "log"
    "net/http"
    "sync/atomic"
    "time"
)

func main() {
    count := uint64(0)
    tick := time.NewTicker(time.Second)
    http.ListenAndServe(":6060", http.HandlerFunc(
        func(w http.ResponseWriter, r *http.Request) {
            atomic.AddUint64(&count, 1)
            select {
            case <-tick.C:
                log.Printf("Rate: %d", atomic.SwapUint64(&count, 0))
            default:
            }
        },
    ))
}

[ghost]

Very nice.

This is the command I am running:

echo "GET http://localhost:6060" | ./vegeta attack -duration=60s -plan="50@0s 100@20s 200@30s" > results.bin

This is the log

2016/02/08 11:35:54 Rate: 0
2016/02/08 11:35:54 Rate: 17
2016/02/08 11:35:55 Rate: 49
2016/02/08 11:35:56 Rate: 49
2016/02/08 11:35:57 Rate: 49
2016/02/08 11:35:58 Rate: 49
2016/02/08 11:35:59 Rate: 49
2016/02/08 11:36:00 Rate: 49
2016/02/08 11:36:01 Rate: 49
2016/02/08 11:36:02 Rate: 49
2016/02/08 11:36:03 Rate: 49
2016/02/08 11:36:04 Rate: 49
2016/02/08 11:36:05 Rate: 49
2016/02/08 11:36:06 Rate: 49
2016/02/08 11:36:07 Rate: 49
2016/02/08 11:36:08 Rate: 49
2016/02/08 11:36:09 Rate: 49
2016/02/08 11:36:10 Rate: 49
2016/02/08 11:36:11 Rate: 49
2016/02/08 11:36:12 Rate: 49
2016/02/08 11:36:13 Rate: 49
2016/02/08 11:36:14 Rate: 67
2016/02/08 11:36:15 Rate: 99
2016/02/08 11:36:16 Rate: 99
2016/02/08 11:36:17 Rate: 99
2016/02/08 11:36:18 Rate: 99
2016/02/08 11:36:19 Rate: 99
2016/02/08 11:36:20 Rate: 99
2016/02/08 11:36:21 Rate: 99
2016/02/08 11:36:22 Rate: 99
2016/02/08 11:36:23 Rate: 99
2016/02/08 11:36:24 Rate: 137
2016/02/08 11:36:25 Rate: 199
2016/02/08 11:36:26 Rate: 199
2016/02/08 11:36:27 Rate: 199
2016/02/08 11:36:28 Rate: 199
2016/02/08 11:36:29 Rate: 199
2016/02/08 11:36:30 Rate: 199
2016/02/08 11:36:31 Rate: 199
2016/02/08 11:36:32 Rate: 199
2016/02/08 11:36:33 Rate: 199
2016/02/08 11:36:34 Rate: 199
2016/02/08 11:36:35 Rate: 199
2016/02/08 11:36:36 Rate: 199
2016/02/08 11:36:37 Rate: 199
2016/02/08 11:36:38 Rate: 199
2016/02/08 11:36:39 Rate: 199
2016/02/08 11:36:40 Rate: 199
2016/02/08 11:36:41 Rate: 199
2016/02/08 11:36:42 Rate: 199
2016/02/08 11:36:43 Rate: 199
2016/02/08 11:36:44 Rate: 199
2016/02/08 11:36:45 Rate: 199
2016/02/08 11:36:46 Rate: 199
2016/02/08 11:36:47 Rate: 199
2016/02/08 11:36:48 Rate: 199
2016/02/08 11:36:49 Rate: 199
2016/02/08 11:36:50 Rate: 199
2016/02/08 11:36:51 Rate: 199
2016/02/08 11:36:52 Rate: 199
2016/02/08 11:36:53 Rate: 199

Some cursory analysis suggests that this feature has passed.

[ghost]

Hi @tsenart

Would you like to merge the code with master and I could give a PR with some documentation?

[tsenart]

Hi there,

I need to spend sometime writing tests and thinking about the API before considering this done. I hope you're not blocked by this though. On Wed, 10 Feb 2016 at 11:19, aidylewis notifications@github.com wrote:

Hi @tsenart https://github.com/tsenart

Would you like to merge the code with master and I could give a PR with some documentation?

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-182295394.

[ghost]

Not blocked. Give me a shout, if I can help with anything, as I am becoming more comfortable with Go code. Thank you.

[tsenart]

So, after having this problem loaded in my mind, and taking a step back, I realised that it's really easy to replicate this feature with a very very simple shell script. For instance:

#!/bin/bash

set -euo pipefail

for rate in 50 100 150 300; do
  echo "GET http://:6060" | ./vegeta attack -rate="$rate" -duration=5s > "results-$rate.bin"
done

./vegeta report -inputs="$(echo results-*.bin | tr ' ' ',')"

@aidylewis: Do you see a reason why you'd want to have this feature in Vegeta still? I think the extra code complexity and the code API breakage offsets the convenience.

[ghost]

I really like this approach, but how would we associate a variable length of duration with rate?

[tsenart]

@aidylewis: Here are two different ways.

#!/bin/bash

set -euo pipefail

rates=(50 100 150 300)
durations=(2s 4s 6s 8s)

for i in $(seq 0 3); do
  echo "GET http://:6060" | vegeta attack -rate="${rates[$i]}" -duration="${durations[$i]}" > "results-${i}.bin"
done

vegeta report -inputs="$(echo results-*.bin | tr ' ' ',')"

#!/bin/bash

set -euo pipefail

echo "GET http://:6060" > targets.txt
vegeta attack -rate=50 -duration=2s > results.1.bin 
vegeta attack -rate=100 -duration=4s > results.2.bin
vegeta attack -rate=150 -duration=6s > results.3.bin
vegeta attack -rate=300 -duration=8s > results.4.bin
vegeta report -inputs="$(echo results.*.bin | tr ' ' ',')"

[ghost]

Very good. The ability to pass in multiple result bins to the report is a real winner. Thanks for your help.

[tsenart]

May I consider this issue closed or do you think it'd be worth documenting this method more thoroughly?

[ghost]

Hi Tomas,

Yes, I would consider this now closed. I will submit a PR with some documentation in the morning and you can tell me what you think. Many Thanks.

[ghost]

Hi @tsenart

I have been away for a couple of days. I don't mind writing documentation on this (e.g. a 'Tips' wiki), but users can also search the repo and get your answers via these threads. WDYT?

[tsenart]

A tutorials / usage wiki would be amazing! Sounds good to me :-) On Wed, 17 Feb 2016 at 10:40, aidylewis notifications@github.com wrote:

Hi @tsenart https://github.com/tsenart

I have been away for a couple of days. I don't mind writing documentation on this (e.g. a 'Tips' wiki), but users can also search the repo and get your answers via these threads. WDYT?

— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-185119068.

[ghost]

I forked the project and started a wiki, but the Wiki changes don't seem to show in the git logs.

Here is what I have anyway: https://github.com/aidylewis/vegeta/wiki/Tips-and-Tricks

[kazimir-malevich]

Was -plan deprecated?