Questions about buckets and rate

[binbin0325]

Question

See below ， As I understand it, rate is making 7,499 requests per second，Why isn't buckets 7499 per second ？

[tsenart]

@fxkr Wanna take this one?

[fxkr]

Because the bucket values aren't requests/second, they're counts of requests over the entire duration of the test.

See... sum of bucket values 21008 + 16630 + 37349 is 74987, which exactly matches the "requests" field in the JSON. (And duration in ns * rate in hz = requests also matches.)

We could have made the bucket values a request rate (by dividing the count we have now by the duration). But a) that's an extra processing step that can easily be done later if needed but can't easily be reverted (divisions are lossy), and b) it would be misleading because it would imply that requests of a certain latency would come in at a fixed rate, which is not necessarily true, and even if it was we wouldn't know.

Does that make sense?

[binbin0325]

As to buckets, I think they are statistics of requests. When I set buckets to 1s,2s,3s,4s.I find that the count in the bucket is not the same as the count in the rate. @fxkr

[fxkr]

As to buckets, I think they are statistics of requests.

I think you still misunderstand what the bucket values are. The buckets contain the total number of requests that fell into the bucket.

This is true:

sum(all bucket values) = requests = rate (in Hz) duration (in ns) 21008 + 16630 + 37349 = 74987 = 7499.366393697744 9999111400e-9

When I set buckets to 1s,2s,3s,4s.I find that the count in the bucket is not the same as the count in the rate.

I think you expect that this should be true:

sum(all bucket values) = rate 21008 + 16630 + 37349 = 7499

But this is false, and that is not a bug. The code does exactly what I intended (for reasons given in my first comment).

The important sentence in the documentation you quoted (which I wrote, just a few days ago) is:

"The values are counts of how many requests fell into that particular bucket."

The documentation matches what the code does. I'm wondering if there's any way to rephrase the documentation to make it clearer? Do you have an idea for a new wording that says the same but that you (and, by extension, hopefully other people too) find clearer?

There is one additional potential problem with your second post. You said "when I set buckets to 1s,2s,3s,4s". You probably just left it out here, but I want to explain this for sake of completeness: you should add a 0 bucket. Otherwise you'll miss all requests that complete faster than 1s.

[binbin0325]

Thanks very much,I think i got it.

[fxkr]

@tsenart next!

[binbin0325]

I'm sorry ，I hava other question，When I set Duration to 1s,Rate to 30, buckets to 0ms,500ms,1s,2s,3s, I see that 20 or more requests fall into the 3s bucket. I think,When I set Duration to 1s, no request should fall into a bucket longer than 1s。 I don't know if I'm getting it right @fxkr @tsenart

[fxkr]

Duration only specifies how long requests are sent for. Vegeta allows every started request the full timeout duration, even if that request was started "just before" the end of the test period. If vegeta didn't do that it would introduce statistical error. If your server doesn't keep up with all requests and lets some time out, with your config, they'll end up in the highest bucket, which is the overflow bucket "3 sec - infinity".

From the readme:

-timeout duration
        Requests timeout (default 30s)

[binbin0325]

So the time spent in the bucket is calculated in terms of the time it takes for the interface to return the response, right?Can I understand it this way

[fxkr]

I think you mean the right thing, but you're not using the right words.

• "time spent in the bucket" - Requests don't spend time in a bucket. Requests are run. Vegeta measures how long they take to complete (either by success, failure or timeout). The difference between start time and completion time determines which bucket the request belongs to. That bucket's counter is increased by 1.

• "interface" - I assume you mean server.

• "return the response" - The server returning a response (success or error) is not the only thing that can happen. There is another thing that can happen: The server could also fail to reply at all. In this case vegeta would eventually treat the request as a timeout (after the configured timeout is over).

Let me try to explain the same in yet another way. This is not quite how it's implemented, but basically this is all vegeta does:

vegeta_start = now()

while now() < vegeta_start + DURATION:

    sleep(1 / RATE)

    start a new background thread that does:
        request_start = now()
        send request
        while no reply received, and now() < request_start + TIMEOUT:
            keep waiting
        request_time = now() - request_start
        bucket[request_time] = bucket[request_time] + 1
        end of background thread

wait until all background threads complete
print buckets

[binbin0325]

Thanks a lot，I got it