na--
commented
4 years ago I'll give an example why I think points 1 and 2 above cover 80-90% of the use cases. Here's how, if we add the startAfter property and the composite executor, you can make an entry file test-suite.js that would allow you to run multiple independent tests:
import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";
export let options = {
execution: {
"test1": {
type: "composite",
exec: "test01.default",
options: test01.options,
},
"test2": {
type: "composite",
exec: "test01.default",
options: test02.options,
startAfter: "test1",
},
"test3": {
type: "composite",
exec: "test02.default",
options: test02.options,
startAfter: "test2",
},
}
}You can execute any of the individual scripts like k6 run test01.js, and you can also execute all of them via test-suite.js. Of course, test-suite.js can be much more dynamic, with some environment variables and helper functions you can build much more flexible and complex execution pipelines, if that's required.
And you'd be able to do something similar even without the composite executor, though it will be a bit fiddly to construct the new options and you'd be restricted to only having a single executor in each sub-test.
imiric
mstoykov
Nicole1991
daniel-kun
kevzettler
loganknecht
kkriegkxs
StanislavDonev
rilham97
UjjwalSk
I'm creating this issue as a place to discuss possible ways for k6 to execute multiple distinct scripts in a single run. This is prompted by https://community.k6.io/t/running-10-test-cases-out-of-100/448, and other similar previous user inquiries.
Something like this can currently be achieved by a simple shell script that sequentially calls
k6 run "$script_path"for all files in a folder, but that workaround is not very convenient without an external output like InfluxDB. It also doesn't work well withk6 cloud, and isn't efficient in general (each separatek6 runhas to re-initialize its VUs). GitLab’s performance testing tool is probably also worth investigating...I also have a vague idea how we can implement something like "test suites" natively in k6 after #1007 is done, and do it relatively easily (:tm: :sweat_smile: ). To get 80% of the way there, we just need to implement one minor (sort-of-already-planned) improvement on the currently planned #1007 feature set :tada:. Instead of specifying
startTime: "30s"for an executor, users should be able to specifystartAfter: "someOtherExecutorName". This will be very easy to implement (minor changes in the config validation, here and here).This will add a lot of convenience in general, so it's probably worth doing just for that. But it will also, when used with the
execoption for each executor (to run some non-defaultfunction), probably satisfy the "test suite" requirements of the majority of our users, albeit with some manual configuration work (which we can ameliorate with some JS helper functions).At a later point, to satisfy the remaining minority of more complex use cases, and to achieve something similar to
k6 run test1.js test2.js test3.js(and, crucially,k6 cloud test1.js test2.js test3.js), we can make another (somewhat more tricky) change to k6. We need to transform the currentlocal.ExecutionSchedulerinto just another executor. That is, we will add a newcompositeexecutor type that will be able to contain other executors. I may be wrong, but because of the new executor architecture in k6, and because we already have 95% of the required code inlocal.ExecutionScheduler, I think this will actually be very straightforward to do :tada: :smile: At least on theexecutionside, there will be some complexity elsewhere.To illustrate, for example, if
test1.jshasvus: 20, duration: "30s"in its exportedoptions,test2.jshasiterations: 1000, vus: 20, andtest3.jshas something like this:then, when you run
k6 run test1.js test2.js test3.js, it should be relatively easy to produce something like this as the final derivedoptions.executionvalue:Notice how, with a few minor changes, we have a whole new layer of flexibility - test suites, for lack of a better term. All the wile, we're preserving the nice new properties of being able to tell precisely how many VUs our test suite will use at most (
20in this case) and what the max test suite duration will be, giving us predictability and easy billing.And, as I mention in the code comment above, the tricky bit isn't actually the execution. It would actually be things like figuring out how to reference the different default functions in the config. I don't think it's a huge issue, but we should be careful how we structure the archive bundles and the related things.
Also, because of the super-global k6 configuration (the curse of #883 strikes again :man_facepalming: ), I'm not sure how we should handle non-execution
optionsthat are different between scripts. We can probably refactor that at a later point though. And things like the planned new HTTP API (issue TBD :blush: ) would greatly reduce the issue... Still, some refactoring is likely required, even for a MVP version of the "full test suites 100% version"...So, yeah, this seems reasonable, after we release #1007: 1) Add a small PR that implements
startAfter, cover (hopefully) 80% of use cases 2) Add a medium-sized PR that implements thecompositeexecutor. No support fork6 run script1.js script2.jsat this point, users have to manually configure the hierarchical config). Hopefully that covers 90% of use cases. 3) Figure out the tricky bits of implementingk6 run script1.js script2.js. This will give us great UX and hopefully 100% coverage of use cases. We can do this step much, much later than steps 1 and 2. Probably smart to do it after we have the new HTTP API.