mstoykov
commented
1 year ago TL;DR(s):
- Extremely unlikely to happen on top of
__ENV. k6/executionsolves this concrete case.- The proposed solution is racy.
- Extension(s) is a better solution for anyone not caring about distributed execution.
Comments
give users the ability to opt in to having
__ENVactually refer to a host machine's environment variables -- perhaps adding that doing so voids your warranty and that this mode is ignored for cloud test runs.
Having to have features that suddenly just no longer work in the cloud or in any kind of distributed fashion seems very bad and not inline with a tool for load testing and k6 specifically.
While we might have not worked very directly on distributed execution it is very much being worked on. There are just a bunch of different pieces that need to be done in order for that to be viable. And in practice a lot of those have been fixed or are being fixed. Still it is some time away, but ...
Adding this will make distributed execution less viable and consistent.
host machines' environment variables
If you are talking about changing the env variables in the outside process running k6 - I am not certain we can do that, and I am definitely against it.
So I would presume you mean what k6 things are the env variables.
At this point you are asking why is __ENV not ... special I guess. As currently it is populated with env variable for each VU and then it just another global variable. As the different VUs run on different JS VMs - they also happen to have different global variables.
This is a bit more complicated yet as __ENV is populated with not only the env variables and what you have provided as --env but also what is defined per scenario. So even if we make them magical now the question is does overriding a scenario defined env be set for all VUs or only the one in the scenario. Should scenario VUs be separated on __ENV to begin with? We can probably get some answers to those, but I would argue any answer will be ... troublesome and likely a thing someone else will argue should've been done the other way around.
Background
I am no expert but AFAIK:
-
node basically does inter process communication between node instances - which arguably is what distributed execution will have to do some extent. And will be a bit more involved than this.
-
Browsers have some ways to communicate between different pages (which is usually where the different JS VMs map to) but even there it is way more complicated than what you propose.
In V8 there's a myriad of ways to pass data
Are those ways between two concurrently running V8 VMs?
K6 has neither of those things and given the nature of the product (multiple, concurrent processes representing virtual users or sessions) it would benefit greatly, imho, from a way for different VUs to send and receive messages down the line - which should avoid many of the pitfalls and complexity involved in managing state across VUs and having too much mutability across different parts of the API by putting that problem squarely in the hands of the user (fine by me!).
I am not disagreeing with you here, but I am very much against shoe horning this on top of an already established API that does not have this semantics and in practice never had them.
And arguably all of the examples you have given are basically not doing anything as strange as letting you write to a variable as it is just a normal variable and then seeing that in what is in practice a different process as if the second process have changed it.
k6/execution can be used in the concrete case.
Real Life Use Case
Looking at the whole example it seems like a somewhat more complicated case for k6/execution.
I am not certain I got the whole code as .. well you skipped the __ENV part entirely and then you also do not show the actual test 🤷♂.
But if each iteration does 1 request you can do: (you can also do it for any constant amount of requests)
// edit: the below counter was wrong in a earlier version.
let globalIndex = exec.scenario.iterationInTest; // import exec from "k6/execution" is required
let product;
for (let vendor in verdors) {
if (vendor.products.length*2 > globalIndex) {// the 2 comes from you doing 2 requests with each product;
globalIndex -= vendor.products.length*2;
continue;
}
product = vendor.products[globalIndex/2]; // same reason for the 2
}
if (product === undefined) {
// we ran out of products
// you can loop again to start from the beginning
}
// rest of your code;- I might be off by 1 somewhere, but shouldn't be hard to fix it.
- you can optimize the search from the vendors by storing the last vendor index you have hit and the last globalIndex and go from there. I doubt this matters a lot for small SharedArrays
- even better optimization will be if you create (in javascript when creating the SharedArray) the sharedArray as only the products and then having the vendor as part of them. Or having SharedArray of vendors and products. This way you can jump directly in the products.
- This works in cloud, and will work in distributed execution like using a k6-operator.
- I would also argue is likely easier to "rewrite" all the problems people are solving with mutable state in cases where you can just use k6/execution + SharedArray. Then it is to make the mutable state actually work.
Technical/syntax problem:
__ENV.somevar++; is NOT an atomic operation which means that you have the classical race condition unless we also introduce:
- mutexes
- atomics, which will be different from the SharedArrayBuffer atomics in ECMASscript
- not actually using
++
Other solutions:
In practice, we can add such an API ... but I doubt it. But I am pretty sure it won't be put on top of `__ENV.
You can always:
- use
k6/exucutionas I show above and transform the issue in going through arrays. - run redis use
k6/redis- which is involved, but likely not as much. - use/build an extension - https://github.com/dgzlopes/xk6-kv seems around the correct place ... but a bit too much for your case. The probably just enough for you https://github.com/mstoykov/xk6-counter is done by me as an example. But is basically
iteartionInScenariobut you can increment it on your own instead of doing it each iteration.
Actually the fact that there is not an extension for this already seems to mean to me it is even less common as what I originally thought - as users have created extensions for all kind of stuff.
I guess 4. for this particular case I would argue you can just go with using the http.asyncRequest and do everything asynchronously. You can then have 1 VU that does multiple requests asynchronously.
The downside is that in order for that to make requests in parallel you will need to make multiple requests in the same iteration - preferably going through the whole Array in one k6 iteration.
Conclusion:
I am going to leave this open so we can have some discussion, and I expect that there will be other proposed solutions, so we can have them here.
GrayedFox
Brief summary
Environment variable changes aren't picked up across different VUs.
Before you hang me!
I've searched around the forums and I understand that, right now, one of the larger issues plaguing the community is one of semantics: naming things is hard and
__ENVcan be more correctly though of as a shorthand way of turning the host machine's environment variables into script parameters - something that the nameENVdoes not belay, thus making it a huge misnomer.But being able to read and write to a host machines environment vars has many valid use cases and is something I hope the K6 team will at least consider - hence this report and some ranting and raving as to why 🙏🏾
I could also phrase this as a feature request (that might have been better) like so: give users the ability to opt in to having
__ENVactually refer to a host machine's environment variables -- perhaps adding that doing so voids your warranty and that this mode is ignored for cloud test runs.k6 version
k6 v0.44.0 (2023-04-24T10:36:01+0000/v0.44.0-0-g14d80f6f, go1.20.3, linux/amd64)
OS
Ubuntu 22.04.2 LTS
Docker version and image (if applicable)
No response
Steps to reproduce the problem
Can be reproduced by using either of the following executors:
I thought this issue was to do with both VUs being instantiated at the same time but I've added a sleep to represent varied response times during a test. Here is some example output:
As we can see, the first VU runs 13 times and the second VU 7 times (expected given the sleeps) for the shared executor while the per vu executor ensures each VU runs 10 times each (also as expected). What surprised me was that changes to the __ENV variable do not affect the next iteration of other VUs.
Expected behaviour
TLDR: I would expect that changes to an
__ENVvar are picked up by different VUs on the same host machine for the next iteration.I would expect output like this (for the shared executor):
Ideally the
__ENVchange is picked up by each VU iteration allowing for information to be shared accross VUs. Sharing via a SharedArray isn't possible due to it being immutable.Background
1722 and #2370 seem to indicate a desire to move away from mutable environment variables and instead treat these values as script parameters (which is sort of how they work at the moment anyway).
This is a significant departure from how environment variables work in different ecosystems - I know K6 isn't Node or V8, nor should it try to be - but there's a lot to be gained from allowing test writers to change environment variables and furthermore ensuring those changes are picked up by different VUs on the same host machine.
In Node, developers can specify whether or not they want to execute something as a child_process or not, which will inherit any environment variables of the parent process. This allows for fine tuned control over environment variables, including child processes being sandboxed so that env changes them don't pollute the global scope - but they are also able to send messages to the parent process which can change an environment variable and/or propagate those changes to other child processes.
In V8 there's a myriad of ways to pass data around and the browser itself doesn't have any direct access to environment variables (but bundlers like webpack can read them during transpile time). Environment variables inside a browser aren't really a thing and there's closures and classes now to take care of encapsulation and a myriad of ways to send messages and move data around and communicate state.
K6 has neither of those things and given the nature of the product (multiple, concurrent processes representing virtual users or sessions) it would benefit greatly, imho, from a way for different VUs to send and receive messages down the line - which should avoid many of the pitfalls and complexity involved in managing state across VUs and having too much mutability across different parts of the API by putting that problem squarely in the hands of the user (fine by me!).
But I digress and that's all a bit of a pipe dream for now. I understand that a core tenant of the K6 design philosophy is repeatability and this includes thinking about tests running in the k6 cloud which may very well end up on being run on separate physical devices.
Actual behaviour
TLDR:
__ENVchanges are only visible inside the VU that makes the change.Real Life Use Case
There are many users, like myself, that use K6:
What this boils down to is this: I would wager a very large swath of your users run their K6 test suite from a single physical device and would therefore benefit from having a way to allow VUs to communicate even if it is by some rudimentary means like by being able to read and write to the host machine's environment variables.
Take the following example JSON representing actual test data. It's read into a K6 readonly
SharedArray:Now this is a small snippet of a JSON file containing 1k lines or so. To make the math easy, let's say we have 50 individual products split uniformly across 10 vendors, i.e. as in the above example, so that each vendor has 5 products.
Thing is we use each product for each vendor twice and want to measure trends of individual vendors - this isn't a stress or performance test. Ideally we would want a shared executor, using 2 or more VUs, making sure that the script file has a few sleeps sprinkled in. The iterations would be capped at 100.
By allowing the VUs to see changes to the host machines environment variables I could handle all of that logic myself by doing something like this (example uses TypeScript):
This would achieve many things:
SharedArrayreadonly struct remains readonly but becomes vastly more useful, solving not only memory consumption issues, but allowing test writers to conditionally feed data into tests without tightly coupling their code to an underlying VU instance ID (indeed, without requiring the k6/execution module at all)Note the
TestRunDatareferences are to getters and setters which read or write to__ENV- implementation is hidden but it's very straight forward. It doesn't work, but imho it should.