grafana / k6

A modern load testing tool, using Go and JavaScript - https://k6.io
GNU Affero General Public License v3.0
25.06k stars 1.24k forks source link

Support for custom non-linear interpolation/progression functions in stages #2256

Open oleiade opened 2 years ago

oleiade commented 2 years ago

Feature Description

In the current state of things, when stages are defined in a load test configuration (options); the interpolation observed during both ramping up and ramping down is always linear.

K6 Linear Ramping up and down

We think, and heard from our users, that it could also be useful to support custom non-linear interpolation functions to drive the transition between stages. Their usage could be driven by a dedicated interpolation in the stages objects.

K6 Ramping up and down

As a starting set of custom interpolation function, we propose:

Suggested Solution (optional)

No response

oleiade commented 2 years ago

In the purpose of documenting the issue, [here's a collection of interpolation functions] that are commonly used in the context of animation/video-games.

In this context, an interpolation function is of the form: func interpolation(startValue, endValue int, delta time.Duration), where startValue and endValue in our case would be the starting and target ramping goals, delta would be the time elapsed since the beginning of the ramping stage (compared to the total duration of the ramping as defined in the options).

oleiade commented 2 years ago

ref #2255

na-- commented 2 years ago

As a starting set of custom interpolation function, we propose: ...

I think we don't need to support all of these functions in the initial implementation. We can start with just 2 or 3, e.g. the current linear default and maybe @LaserPhaser's instant one. In my mind, the more important things are these:

Another reason I don't want to focus on the other more complicated interpolation/progression functions right now is that I'm not sure we should just have a few ones like exponential, easeIn, etc. These are not single functions, they are whole classes of functions that can have different coefficients... Also see https://easings.net/ for some examples of the many possible different types of easing functions. Probably not all of these make sense in the context of load testing, but it might both be more simple and more powerful to support some simple mathematical notation instead of a bunch of inflexible labels... :man_shrugging:

For example, if we allow users to define a function that accepts an argument t with values between 0 and 1 (the percentage of the elapsed time since the start of the stage) and returns a result value r also between 0 and 1 (the percentage of the difference between the start rate/vus of the stage and its target), if should be relatively simple to express any interpolation/scaling function:

But this can get very complicated very quickly, so it's probably better left for the future.

oleiade commented 2 years ago

My initial gut feeling proposal would be to have a dedicated:

type Interpoler interface {
  Interpolate(start, end int64, delta time.Duration) int64
}

as well as various implementations for the functions we want to support:

type LinearInterpolation struct {}

func (li *LinearInterpolation) Interpolate(start, end int64, time.Duration) float64 {
  // Compute value for s(t)
}

(I'm not hung up on the implementation or interface itself)

However, I'm not sure if I understand what you meant when you say "For example, if we allow users to define a function that accepts an argument t with values between..." @na-- Do you mean that instead of an interpolation field in the options, you would allow a user to define/copy a (JS) function there directly?

na-- commented 2 years ago

:man_shrugging: about the name, I don't have any strong opinions here. @MStoykov, @codebien, @yorugac, WDYT about interpolation vs. progression vs. something else?

However, I'm not sure if I understand what you meant when you say "For example, if we allow users to define a function that accepts an argument t with values between..." @na-- Do you mean that instead of an interpolation field in the options, you would allow a user to define/copy a (JS) function there directly?

I think it's probably too premature to discuss these details if we don't plan to implement it soon, but I think we should allow for a simplified mathematical notation here, similarly to how we do so for thresholds, though definitely without any JS code involved, just simple math formulas. Something like this

stages: [
    {target: 100, duration: '1m', progression: 'linear' }, // allow simple shortcut labels for simple cases
    {target: 200, duration: '1m', progression: 'r=t' }, // this is the same as the line above
    {target: 300, duration: '1m', progression: 'instant' }, 
    {target: 400, duration: '1m', progression: 'r=1' }, // this is also the same as the line above
    {target: 500, duration: '1m', progression: 'r=t*t' }, // quadratic ease in
    {target: 600, duration: '1m', progression: 'r=1-(1-t)*(1-t)' }, // quadratic ease out
    // ...
 ],

My initial gut feeling proposal would be to have a dedicated:

type Interpoler interface {
  Interpolate(start, end int64, delta time.Duration) int64
}

I don't think this is very flexible, we probably shouldn't work with absolute values, neither for time nor for values. If both the argument and the result are percentages (representing parts of the stage time and the stage target), the interpolation functions should be completely agnostic and usable both for VUs and for arrival rates, with each executor doing the final conversion itselft. And it will be easier to write, since you won't care about absolute values :man_shrugging: So, something like this:

type Interpolator func (t float64) (result float64)
func LinearInterpollation (t floa64) float64 { return t; }
func InstantInterpollation (_ floa64) float64 { return 1; }

t is the current time in the stage execution as a part of the total stage duration, with values between 0 and 1, and r is the percentage of the current value (either VUs or arrival rate) of the difference between the stage's starting value and its target. In the example above, the calculation at time=30s will be like this:

t = 30s/60s = 0.5 // i.e. halfway through the first stage
r = LinearInterpollation(t) = LinearInterpollation(0.5) =  0.5 // i.e. halfway through the 

ActualVUs/ActualRate = StartVUs + (r * (stage.target - StartVUs)) = 0 + (0.5 * (100 - 0)) = 0.5 * 100 = 50 
// i.e. at time=30s, we expect to have 50 VUs actively running, or 50 iterations/s, depending on the executor
yorugac commented 2 years ago

On names: interpolation seems like the best choice right now. transition is a good fit for animation but it's not what will actually be happening in k6.

Also, it seems to me that from user's perspective the speed of increase and option to change that speed is more important than complex math formulae. E.g. there is apparently a term "peak testing" used in some places, as a very fast increase which corresponds to the real traffic problem. Given that, as a user I'd prefer to have a linear with exposed coefficients or just specifying the formula as in @na--'s example, more than different types of interpolations that would more than likely require one to lookup / understand / remember how they work before usage.

Of course, exponential and others make sense as a shortcut for common-usage patterns: so that people don't have to write the formula each time by hand. So I'm really curious about what other people think about usage patterns, outside of our team too :slightly_smiling_face:

mstoykov commented 2 years ago

First of all, I don't want to go through discussing and making decisions if we are not actually going to be implementing this in "full". IMO if we won't implement (to some extend) user-defined functions then there are really good chances that in the future we will need to rewrite the whole thing.

Having said that I also am kind of against (at least at the beginning) having names such as linear or instant. For me, all of this will just be one more thing that we will need to document so users find and use and if we are already writing documentation we can just as well just give them the functions :shrug:.

Further given the exact place this code stands this will definitely need a lot of benchmarks and due diligence to not worsen performance. I am actually kind of surprised by the fact that the additional switch doesn't make this way worse although it might be due to the benchmarks we have :shrug:

benchmarks

``` name old iterations/s new iterations/s delta RampingArrivalRateRun/VUs10-8 250k ± 3% 253k ± 3% ~ (p=0.247 n=10+10) RampingArrivalRateRun/VUs100-8 306k ± 3% 312k ± 2% ~ (p=0.052 n=10+10) RampingArrivalRateRun/VUs1000-8 302k ± 2% 299k ± 1% -0.94% (p=0.034 n=8+10) RampingArrivalRateRun/VUs10000-8 279k ± 2% 274k ± 1% -2.03% (p=0.000 n=10+8) name old alloc/op new alloc/op delta RampingArrivalRateRun/VUs10-8 69.3MB ± 3% 69.1MB ± 2% ~ (p=0.853 n=10+10) RampingArrivalRateRun/VUs100-8 73.7MB ± 3% 74.9MB ± 2% +1.66% (p=0.043 n=10+10) RampingArrivalRateRun/VUs1000-8 73.4MB ± 3% 73.1MB ± 1% ~ (p=0.182 n=9+10) RampingArrivalRateRun/VUs10000-8 78.7MB ± 2% 77.3MB ± 0% -1.80% (p=0.000 n=9+7) Cal/1s-8 480B ± 0% 648B ± 0% +35.00% (p=0.000 n=10+10) Cal/1m0s-8 480B ± 0% 648B ± 0% +35.00% (p=0.000 n=10+10) CalRat/1s-8 6.39MB ± 0% 6.39MB ± 0% ~ (p=0.481 n=10+10) CalRat/1m0s-8 714MB ± 0% 714MB ± 0% ~ (p=0.243 n=10+9) RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8 246kB ± 0% 246kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8 2.65MB ± 0% 2.65MB ± 0% -0.00% (p=0.006 n=8+10) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8 246kB ± 0% 246kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8 2.65MB ± 0% 2.65MB ± 0% -0.00% (p=0.003 n=6+9) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8 73.7kB ± 0% 73.7kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8 795kB ± 0% 795kB ± 0% -0.00% (p=0.013 n=8+10) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8 24.6kB ± 0% 24.6kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8 270kB ± 0% 270kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8 98.3kB ± 0% 98.3kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8 1.06MB ± 0% 1.06MB ± 0% ~ (p=0.973 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8 90.1kB ± 0% 90.1kB ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8 983kB ± 0% 983kB ± 0% ~ (all equal) VUHandleIterations-8 1.61kB ± 4% 1.62kB ± 3% ~ (p=1.000 n=10+10) name old allocs/op new allocs/op delta RampingArrivalRateRun/VUs10-8 1.44M ± 3% 1.44M ± 2% ~ (p=0.853 n=10+10) RampingArrivalRateRun/VUs100-8 1.53M ± 3% 1.56M ± 2% ~ (p=0.052 n=10+10) RampingArrivalRateRun/VUs1000-8 1.53M ± 2% 1.51M ± 1% -0.92% (p=0.027 n=8+10) RampingArrivalRateRun/VUs10000-8 1.57M ± 1% 1.54M ± 0% -1.87% (p=0.000 n=9+7) Cal/1s-8 2.00 ± 0% 4.00 ± 0% +100.00% (p=0.000 n=10+10) Cal/1m0s-8 2.00 ± 0% 4.00 ± 0% +100.00% (p=0.000 n=10+10) CalRat/1s-8 44.4k ± 0% 44.4k ± 0% ~ (p=0.318 n=10+10) CalRat/1m0s-8 2.91M ± 0% 2.91M ± 0% ~ (p=0.123 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) VUHandleIterations-8 15.4 ± 4% 15.5 ± 3% ~ (p=1.000 n=10+10) name old time/op new time/op delta Cal/1s-8 5.00µs ±14% 5.80µs ±16% +15.91% (p=0.004 n=10+9) Cal/1m0s-8 307µs ± 3% 303µs ± 6% ~ (p=0.200 n=9+8) CalRat/1s-8 14.9ms ± 6% 14.6ms ± 3% ~ (p=0.436 n=9+9) CalRat/1m0s-8 8.82s ± 5% 8.83s ± 5% ~ (p=0.604 n=10+9) RampingVUsGetRawExecutionSteps/seq:;segment:/normal-8 364µs ± 8% 358µs ± 7% ~ (p=0.497 n=10+9) RampingVUsGetRawExecutionSteps/seq:;segment:/rollercoaster-8 3.74ms ± 5% 3.82ms ± 7% ~ (p=0.297 n=9+9) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/normal-8 315µs ± 6% 363µs ± 8% +15.40% (p=0.000 n=9+10) RampingVUsGetRawExecutionSteps/seq:;segment:0:1/rollercoaster-8 3.46ms ± 9% 3.67ms ±10% +6.26% (p=0.023 n=10+10) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/normal-8 105µs ± 9% 105µs ± 7% ~ (p=0.853 n=10+10) RampingVUsGetRawExecutionSteps/seq:0,0.3,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.3/rollercoaster-8 1.15ms ± 6% 1.14ms ± 6% ~ (p=0.436 n=10+10) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/normal-8 34.8µs ±17% 35.9µs ± 9% ~ (p=0.684 n=10+10) RampingVUsGetRawExecutionSteps/seq:0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1;segment:0:0.1/rollercoaster-8 371µs ± 5% 386µs ± 7% +4.09% (p=0.043 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/normal-8 134µs ± 7% 141µs ± 5% +5.17% (p=0.011 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:2/5:4/5/rollercoaster-8 1.52ms ± 8% 1.54ms ±10% ~ (p=0.529 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/normal-8 135µs ±11% 138µs ± 9% ~ (p=0.579 n=10+10) RampingVUsGetRawExecutionSteps/seq:;segment:2235/5213:4/5/rollercoaster-8 1.35ms ± 7% 1.38ms ± 8% ~ (p=0.393 n=10+10) VUHandleIterations-8 1.00s ± 0% 1.00s ± 0% ~ (p=0.247 n=10+10) name old iterations/ns new iterations/ns delta VUHandleIterations-8 0.08 ± 6% 0.09 ± 8% +6.25% (p=0.019 n=10+10) ```

Even running real tests (with impossible requirements) against master and #1391 seems to show that I get the same (empty) iterations/s around 250k, but here the fluctuation is much bigger. I would still expect that adding more stuff to this will make it slower and also x86 is really good at branch prediction (usually), while something like AWS graviton might not be as good, so I would argue we should probably benchmark this specific code even more. (also we already had problems with ARM around this parts)

On the same note - there is IMO a lot of headroom 250k empty iterations is already an absurdly big amount and will get better if instead of calculating for each iteration we were doing it for number(bucket) of them as mentioned in #1386.

I also do prefer if we use the word function or f as part of the name so either just purely function/f or rampingF rampF rampFunction. Even just ramping="r=t" seems pretty nice, maybe even without the r=, it also goes really well with "linear", "instant" and so on, if/when we add them.

Some ... technical notes:

ramping-vus and ramping-arrival-rate kind of calculate two different things:

Said slightly differently: for the first, the rate is constant throughout a stage, while the other calculates when an iteration should start given that the rate of that changes throughout the stage.

This is also why they are done in two different ways and why in practice what ramping-vus calculates during ramping is what ramping-arrival-rate calculates during ... constant segments ;) (the one just calculates based on the end of the stage, while the other does it based on the start, also some of the variables aren't the same etc. etc.)

What I am trying to say is that the two things aren't the same, so having the same implementation for both will require some amount of mathematics to go between the two ;). And from what I can see it will be only partially used for ramping-arrival-rate

ramping-arrival-rate details

Ramping arrival rate does already use calculus(I need to reword parts of this and probably hide the equation in a collapsable section) to calculate when the next iteration should happen. It does this (as explained there) by calculating the t(ime) at which the area "below" the graph of the particular function will be equal to 1,2,3,4,5 ...

 4|
 3|       ,-+
 2|----+-'  |
 1|    |    |
  +----+----+------->
  0s   5s   10s

^ stolen from the code like in the graph above if we go between 5s and 10s and we go from 2iter/s to 3iter/s. Just going from 2 to 3 over 5s will make (formula for area of a triangle) 5 (3-2) /2 = 2.5 iterations in total, but we also have 2iter/s happening over the whole time which is(area of a rectangle) 25=10 iterations. So for that 5s k6 should start 12.5 iterations, but they do not start at regular intervals what we actually need to calculate is the t after 5s for which the area of the graph between 5s and t, below the graph is equal to 1 and then the same for 2 and so on. This also gets (somewhat) more complicated with execution segments because we might not actually need to do the 1st or 2nd iteration on this particular k6 instance, but instead calculate the 3rd (for example). The more complicated part actually comes from the fact that this goes across stages, not just for the current one so for the next stage (if there is one) there is 0.5 iteration that needs to be "carried over" and also it will be making the 13th iteration for the whole scenario which may or may not be for this one instance and can't just be calculated the same way as if it was the 1st (as it's the 1st for that stage).

This means that while the "ramping" for that section is r=t the actual formula is f(x)=((3-2)/5)(t-5)+ 2 where the 3-2 comes from the start and end of the iter/s, /5 is the time over which we get there, -5 is because our t goes from 5 to 10 so we need to remove the start time and +2 is because we already are doing 2 iterations/s.

Also again this just calculates how many iter/s should be started at any given time not when they should be started. To get the latter we basically need to calculate the integral of that function and then solve it when it's equal to the iteration we want to do next and we get the t.(I really don't want to try to rehash this whole thing again it's explained in both the linked comment and the code). The code for ... simplicity reasons calculates this by "moving" all functions to start at t=0 which removes some of the complexity from the equation (the -5 above ;))

Given all of this, I do think to implement this we will need some more .... experiments and finding some easy (at least computationally) ways to get the integral of a function, which is definitely possible as I have done it in (strawberry) Prolog class .... 10 years ago (this looks familiar, although I have no idea what any of this does :rofl: ) and we do IMO not need to do it for any possible function just for most ones that we care about. We then need to again have a programmable way to "simplify" the equation so we are actually calculating it when it's equal to some number(the iteration) and to get from it the t. Again given that wolfram alpha can do it, it's possible it just will mean it will need some research and I am not particularly certain how performant it will be.

As a whole, this was one of the many reasons why I didn't even start on it back then but just sprinkled some comments in places where it will be good idea to think about it in the future.

na-- commented 2 years ago

Ah, yeah, I forgot that for ramping-arrival-rate we actually couldn't just use the raw equation, or calculate its integral directly, but we had to actually solve the integral to get what we need :disappointed: Should be fairly simple for big classes of functions, but the implementation complexity of figuring all it out for such a seemingly minor feature is probably not worth it right now...

I don't see why we can't start with simple "shortcut" labels like linear or instant in the beginning though? They should be fairly simple to implement, and I don't get your arguments about performance implications :confused: The current code would need a bit of refactoring to make it more readable, but there shouldn't be any performance differences when using a few well-known functions

LaserPhaser commented 2 years ago

@na-- @MStoykov

Hello, sorry, but it's not clear to me. Is It worth implementing a solution with `linear/instant' functions or with function support?

Or K6 team is not going to merge any of these solutions?

na-- commented 2 years ago

@LaserPhaser, sorry for the confusion, as you can see, we're not quite sure ourselves yet... :sweat_smile:

@MStoykov and anyone else with an opinion from the @grafana/k6-devs and community, please :+1: or {:-1: + an explanation} this post to signal if you're ok with this as the first step here:

WDYT? On the one hand, it seems like a simple and safe backwards-compatible change. On the other hand, it doesn't really add anything functionally - we already support 0-duration stages that serve the exact same purpose as the proposed interpolation: 'instant' property, just more verbosely...

So I'm on the fence if we should have stages.interpolation without any extra capabilities. But we can't agree on how these extra capabilities (e.g. other interpolation types) should be implemented, and it seems like a relatively low-prio issue while it's apparent there will be a lot of complexity with the ramping-arrival-rate executor. So, I am a firm "maybe" or ":man_shrugging:" here... :sweat_smile:

mstoykov commented 2 years ago

I am against it as I mentioned above. For me this is a shortcut that requires:

  1. a decision on how the property will be called - I dislike interpolation, I still think ramping is better. *
  2. requires a bunch of code changes in different places, some of which are in the executors which are already big enough
  3. will definitely require documentation and even more explanations, including that only those options are available(I do expect a lot of people will come to ask us to implement everything under the sun after this)

The benefit is marginal as IMO we still need to document this so anyone will use it or find it, so we can just as well document the current behaviour.

LaserPhaser commented 2 years ago

Maybe I can just update the documentation in k6-docs rep, with an example how to do steps in load testing. From my opinion It will be fair enough on current moment to explain it to the community or I can add little blog. WDYT?

na-- commented 2 years ago

@LaserPhaser, good point. Regardless of what we decide here, we should document the existing solution, since it seems it isn't mentioned anywhere, so I created https://github.com/grafana/k6-docs/issues/516 for that.

yorugac commented 2 years ago

The "first step" change seems a bit like a syntactic sugar right now but as a potential benefit, it might trigger additional feedback that will in turn help flesh out any further changes here. Though as Mihail mentioned, that might turn bizarre too :smile: