Closed ghost closed 8 years ago
With some thinking to do about the command line interface, I'm in favour of this proposal.
+1
Hi @tsenart
The ramp-up is becoming increasingly important to us. I wondered if you could give me some pointers, so I could attempt to implement this myself?
Hello @aidylewis,
I'm sorry, I haven't had time to work on Vegeta recently. I took 2 hours to implement this feature in the plans
branch. It's not well tested and probably needs some work with a clearer mind to get all the edge cases and quality sorted, but you can check it out, build it, test it, etc :-)
Here's an example invocation:
echo "GET http://:6060" | ./vegeta attack -timeout=6s -duration=20s -plan="100@2s 200@4s 400@6s 500@8s 600@10s" | tee result.bin | ./vegeta report
This is fantastic. We appreciate this so much. I'll test it out tomorrow. Thanks.
For anyone else who would like to test this branch:
go get github.com/tsenart/vegeta
cd $GOPATH/src/github.com/tsenart/vegeta
git checkout plans
go install github.com/tsenart/vegeta
Hi @tsenart
-plan="100@20s 200@20s”
> 2016/02/04 09:15:02 bad timing order in plan "100@20s 200@20s"
Will the below start off at 100 r/s and then after ten seconds ramp-up to 200 r/s?
echo "GET http://whatever" | vegeta attack -duration=30s -plan="100@10s 200@20s" > results.bin
We never make it to 200 r/s.
Requests [total, rate] 3989, 133.03
Duration [total, attack, wait] 29.999565631s, 29.985s, 14.565631ms
I think you misunderstood the semantics. In a plan, you define relative points in time where the rate should change. For instance, "100@10s 200@20s" means that at 10s into the attack, the rate will change to 100 and at 20s into the attack, the rate will change to 200.
The duration is relevant due to these semantics. On Thu, 4 Feb 2016 at 10:56, aidylewis notifications@github.com wrote:
Hi @tsenart https://github.com/tsenart
- We cannot add individuals plans of the same duration.
-plan="100@20s 200@20s”
2016/02/04 09:15:02 bad timing order in plan "100@20s 200@20s"
- How is the ramp-up supposed to work?
Will the below start off at 100 r/s and then after ten seconds ramp-up to 200 r/s?
echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=30s -plan="100@10s 200@20s" > results.bin
We never make it to 200 r/s.
Requests [total, rate] 3989, 133.03 Duration [total, attack, wait] 29.999565631s, 29.985s, 14.565631ms
- Do you think we need a duration flag when we have a plan?
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179739572.
Hi @tsenart
Thanks for your response.
If at 10s into the attack the rate change will be 100, what will be the rate from 0 - 10s?
As it stands, it would be 100, since you didn't specify any previous rate. To do so, you can specify a rate at t=0. e.g. 50@0 100@10s 200@20s
As you know, I did this in two rushed hours. Please give me any feedback you have on the interface, expectations, etc.
The interface is sensible, I just misunderstood it. I'll do some testing and ask @DaveBlooman from News to do some testing/give feedback as they are also interested in the functionality.
In fact, I think we need an interface where we specify the ramp-up over a specific period of time and the constant rate over a specific period of time and remove the duration. But I am not sure how to convert that into command line options.
rampto=200for5m constant=200for15m # which is ugly
Thoughts @daveblooman & @gunisp01?
I don't see how you can't achieve that with the current interface. On Thu, 4 Feb 2016 at 14:27, aidylewis notifications@github.com wrote:
In fact, I think we need an interface where we specify the ramp-up over a specific period of time and the constant rate over a specific period of time and remove the duration. But I am not sure how to convert that into command line options.
rampto=200for5m constant=200for15m # which is ugly
Thoughts @daveblooman https://github.com/daveblooman & @PoornimaGunisetti?
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179834800.
We can make it work with current interface as 50@0 100@10s 200@20s 200@20m
Did you mean 2000@20m? Because if it's 200@20m then there would be no change from 200@20s. On Thu, 4 Feb 2016 at 14:30, Poornima Gunisetti notifications@github.com wrote:
We can make it work with current interface as 50@0 100@10s 200@20s 200@20m
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179836885.
Nope I meant 200@20m only as we need it to ramp up within first few seconds and there after wait for few minutes (constant rate). In case we do not put 200@20s and 200@20m, it will take 20 minutes to ramp up to 200 which is not something we need. Correct me if I am wrong in understanding @aidylewis requirement here!
You can drop the last one. 50@0 100@10s 200@20s
with a large enough
attack duration should do the trick.
On Thu, 4 Feb 2016 at 14:37, Poornima Gunisetti notifications@github.com
wrote:
Nope I meant 200@20m only as we need it to ramp up within first few seconds and there after wait for few minutes (constant rate). In case we do not put 200@20s and 200@20m, it will take 20 minutes to ramp up to 200 which is not something we need. Correct me if I am wrong in understanding @aidylewis https://github.com/aidylewis requirement here!
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179842961.
Ignore me, I just realised it is the time at which "ramp up will occur" rather than "ramping up" time.
Yes, the interface is fine. We'll do some testing.
The attack rate between plan transitions is always constant. So for "50@0 100@10s 200@20s" with an attack duration of 30 min, the rates would be.
0s until 10s: 50 10s until 20s: 100 20s until 30m: 200 On Thu, 4 Feb 2016 at 14:43, Poornima Gunisetti notifications@github.com wrote:
Cool, makes sense. But what if we need constant rate between ramp up? How can we achieve that?
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179846384.
Example:
shell_1
echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=120s -plan="5@0 10@30s 20@60s" > results.bin
shell_2
while true; do cat results.bin | vegeta report ; sleep 2; done
In the above attack, Vegeta is not managing to reach 10 r/s from 30s to 60s and we never reach 20 r/s
How are you verifying this? On Thu, 4 Feb 2016 at 15:44, aidylewis notifications@github.com wrote:
In the above attack, Vegeta is not managing to reach 10 r/s from 30s to 60s and we never reach 20 r/s (even though we have given 60 seconds to achieve this throughput).
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179877022.
Using the above bash loop which pipes the results.bin into the Vegeta report.
Although, I can't see it in the code, you probably average the current total rate, when I am thinking it is the current rate.
No. It's the average rate, not the last one. On Thu, 4 Feb 2016 at 19:37, aidylewis notifications@github.com wrote:
Although, I can't see it in the code, you probably average the current total rate, when I am thinking it is the current rate.
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-179990291.
My fault. I will try tcpflow or something tomorrow.
I am using httpry on a OS X network interface at 1 second intervals.
sudo httpry -i en0 -s -t 1
We don't get to 10 r/s between 30 - 60 secs, and don't reach 20 r/s.
echo "GET http://www.stage.bbc.co.uk/news" | vegeta attack -duration=120s -plan="5@0 10@30s 20@60s" > results.bin
I'm not sure that tool is measuring the right thing right. Here's a short Go HTTP server program which should show you how many requests per second it is receiving.
package main
import (
"log"
"net/http"
"sync/atomic"
"time"
)
func main() {
count := uint64(0)
tick := time.NewTicker(time.Second)
http.ListenAndServe(":6060", http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
atomic.AddUint64(&count, 1)
select {
case <-tick.C:
log.Printf("Rate: %d", atomic.SwapUint64(&count, 0))
default:
}
},
))
}
Very nice.
This is the command I am running:
echo "GET http://localhost:6060" | ./vegeta attack -duration=60s -plan="50@0s 100@20s 200@30s" > results.bin
This is the log
2016/02/08 11:35:54 Rate: 0
2016/02/08 11:35:54 Rate: 17
2016/02/08 11:35:55 Rate: 49
2016/02/08 11:35:56 Rate: 49
2016/02/08 11:35:57 Rate: 49
2016/02/08 11:35:58 Rate: 49
2016/02/08 11:35:59 Rate: 49
2016/02/08 11:36:00 Rate: 49
2016/02/08 11:36:01 Rate: 49
2016/02/08 11:36:02 Rate: 49
2016/02/08 11:36:03 Rate: 49
2016/02/08 11:36:04 Rate: 49
2016/02/08 11:36:05 Rate: 49
2016/02/08 11:36:06 Rate: 49
2016/02/08 11:36:07 Rate: 49
2016/02/08 11:36:08 Rate: 49
2016/02/08 11:36:09 Rate: 49
2016/02/08 11:36:10 Rate: 49
2016/02/08 11:36:11 Rate: 49
2016/02/08 11:36:12 Rate: 49
2016/02/08 11:36:13 Rate: 49
2016/02/08 11:36:14 Rate: 67
2016/02/08 11:36:15 Rate: 99
2016/02/08 11:36:16 Rate: 99
2016/02/08 11:36:17 Rate: 99
2016/02/08 11:36:18 Rate: 99
2016/02/08 11:36:19 Rate: 99
2016/02/08 11:36:20 Rate: 99
2016/02/08 11:36:21 Rate: 99
2016/02/08 11:36:22 Rate: 99
2016/02/08 11:36:23 Rate: 99
2016/02/08 11:36:24 Rate: 137
2016/02/08 11:36:25 Rate: 199
2016/02/08 11:36:26 Rate: 199
2016/02/08 11:36:27 Rate: 199
2016/02/08 11:36:28 Rate: 199
2016/02/08 11:36:29 Rate: 199
2016/02/08 11:36:30 Rate: 199
2016/02/08 11:36:31 Rate: 199
2016/02/08 11:36:32 Rate: 199
2016/02/08 11:36:33 Rate: 199
2016/02/08 11:36:34 Rate: 199
2016/02/08 11:36:35 Rate: 199
2016/02/08 11:36:36 Rate: 199
2016/02/08 11:36:37 Rate: 199
2016/02/08 11:36:38 Rate: 199
2016/02/08 11:36:39 Rate: 199
2016/02/08 11:36:40 Rate: 199
2016/02/08 11:36:41 Rate: 199
2016/02/08 11:36:42 Rate: 199
2016/02/08 11:36:43 Rate: 199
2016/02/08 11:36:44 Rate: 199
2016/02/08 11:36:45 Rate: 199
2016/02/08 11:36:46 Rate: 199
2016/02/08 11:36:47 Rate: 199
2016/02/08 11:36:48 Rate: 199
2016/02/08 11:36:49 Rate: 199
2016/02/08 11:36:50 Rate: 199
2016/02/08 11:36:51 Rate: 199
2016/02/08 11:36:52 Rate: 199
2016/02/08 11:36:53 Rate: 199
Some cursory analysis suggests that this feature has passed.
Hi @tsenart
Would you like to merge the code with master and I could give a PR with some documentation?
Hi there,
I need to spend sometime writing tests and thinking about the API before considering this done. I hope you're not blocked by this though. On Wed, 10 Feb 2016 at 11:19, aidylewis notifications@github.com wrote:
Hi @tsenart https://github.com/tsenart
Would you like to merge the code with master and I could give a PR with some documentation?
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-182295394.
Not blocked. Give me a shout, if I can help with anything, as I am becoming more comfortable with Go code. Thank you.
So, after having this problem loaded in my mind, and taking a step back, I realised that it's really easy to replicate this feature with a very very simple shell script. For instance:
#!/bin/bash
set -euo pipefail
for rate in 50 100 150 300; do
echo "GET http://:6060" | ./vegeta attack -rate="$rate" -duration=5s > "results-$rate.bin"
done
./vegeta report -inputs="$(echo results-*.bin | tr ' ' ',')"
@aidylewis: Do you see a reason why you'd want to have this feature in Vegeta still? I think the extra code complexity and the code API breakage offsets the convenience.
I really like this approach, but how would we associate a variable length of duration with rate?
@aidylewis: Here are two different ways.
#!/bin/bash
set -euo pipefail
rates=(50 100 150 300)
durations=(2s 4s 6s 8s)
for i in $(seq 0 3); do
echo "GET http://:6060" | vegeta attack -rate="${rates[$i]}" -duration="${durations[$i]}" > "results-${i}.bin"
done
vegeta report -inputs="$(echo results-*.bin | tr ' ' ',')"
#!/bin/bash
set -euo pipefail
echo "GET http://:6060" > targets.txt
vegeta attack -rate=50 -duration=2s > results.1.bin
vegeta attack -rate=100 -duration=4s > results.2.bin
vegeta attack -rate=150 -duration=6s > results.3.bin
vegeta attack -rate=300 -duration=8s > results.4.bin
vegeta report -inputs="$(echo results.*.bin | tr ' ' ',')"
Very good. The ability to pass in multiple result bins to the report is a real winner. Thanks for your help.
May I consider this issue closed or do you think it'd be worth documenting this method more thoroughly?
Hi Tomas,
Yes, I would consider this now closed. I will submit a PR with some documentation in the morning and you can tell me what you think. Many Thanks.
Hi @tsenart
I have been away for a couple of days. I don't mind writing documentation on this (e.g. a 'Tips' wiki), but users can also search the repo and get your answers via these threads. WDYT?
A tutorials / usage wiki would be amazing! Sounds good to me :-) On Wed, 17 Feb 2016 at 10:40, aidylewis notifications@github.com wrote:
Hi @tsenart https://github.com/tsenart
I have been away for a couple of days. I don't mind writing documentation on this (e.g. a 'Tips' wiki), but users can also search the repo and get your answers via these threads. WDYT?
— Reply to this email directly or view it on GitHub https://github.com/tsenart/vegeta/issues/163#issuecomment-185119068.
I forked the project and started a wiki, but the Wiki changes don't seem to show in the git logs.
Here is what I have anyway: https://github.com/aidylewis/vegeta/wiki/Tips-and-Tricks
Was -plan deprecated?
In this AWS document: http://aws.amazon.com/articles/1636185810492479, it states: "We recommend that you increase the load at a rate of no more than 50 percent every five minutes. Both step patterns and linear patterns for load generation should work well with Elastic Load Balancing"
I thus propose a ramp-up with an the interface @tsenart has proposed on the project Gitter chat: