Support for Multiple Network Card

[FournierAlexandre]

Hello,

We would like to use K6 on machine that have multiple network card. As for now K6 does not support multiple NIC for their VUS.

What will be the best way of doing it?

Thanks!

Alex

[robingustafsson]

Hi @FournierAlexandre, good suggestion.

There are a couple of things to discuss here as I see it, both implementation and configuration.

Let's start with the user facing, configuration part. Two things to consider here, where to specify it; CLI vs script options (is this something you'd want to version control with the script?), and what format to use to identify the NICs; IP/CIDR vs interface name.

I guess we at least have the following possibilities in terms of CLI (allowing CLI option to be specified more than once):

• CLI and IP: k6 run --nic 1.2.3.4
• CLI and CIDR (matching any NIC with an IP in the block): k6 run --nic 1.2.3.0/24
• CLI and interface name: k6 run --nic eth0 (or k6 run --nic eth+ to match all eth interfaces etc.)

and for script option something along the lines of this:

export let options = {
    nics: ["eth0", "eth+", "!eth", "1.2.3.4", "1.2.3.4/32", "2a00:1450:400f:806::200e"]
};

I'd propose we start with a CLI option (like --nic) supporting both IP addresses, CIDRs and interface names as input.

Now, in terms of implementation would a simple round-robin selection of interface to use for a VU to make TCP connections be enough?

[FournierAlexandre]

Hello @robingustafsson ,

Thanks for your answer!

I really like the option in the script that you can put in Source control. We have different script for different environment.

So I guess both won't be too much since several feature are available in the CLI and the options. It would make sens for such a configuration.

A round-robin will be more than enough, the goal is to use all card available with the same amount of VUS on each if possible. But we could have different strategies that we can specify in CLI or in Options in the future, this way we can implement different strategy if we needed (Round robin, Proportion (20% on a card, 80% on other, etc.)

Are you planning of implementing this feature?

[robingustafsson]

Great, yeah having the options in the script is nice for version control. Having both CLI and options should be the goal, I agree with that.

Having different strategies in the future for how to allocate VUs onto NICs sounds like a good idea, starting with round-robin as the first strategy.

This is a needed feature, but currently not a high priority for us (Load Impact), but PRs are of course welcome. Do I remember correctly hearing you already had some code written for this?

[ofauchon]

Hi.

I started implementation of "multiple nic/ip implementation": http://github.com/ofauchon/k6/tree/feature/476_MultipleNics

Right now , the new option '-n' / '--nic ' will detect ips for the given interface and affects them to VUs randomly .

I'll add multiple interface (-nic eth0,eth1) support soon.

What else would you need before PR ?

[FournierAlexandre]

I am working with the Team that will use it, I will get back to you as soon as I have more information!

Thanks a lot!

[ofauchon]

Great !

Remember it has only be tested on a local developpement environnement at the moment.

If it doesnt work as expected, please send me your setup detail (output of 'ip addr' 'ip route' commands, sample of injection script, and k6 command line with arguments you use)

So I can set up a test lab and reproduce the conditions of your test.

Thanks,

Olivier.

[ofauchon]

Hi,

Today, I could test "-nic eth0" with 100+ differents ip alias during a real performance test. It worked fine for me, and I could see all the 100+ IPs in the logs of targetted servers.

But I'm still wondering if It's the good way to do the job.

Using command line arguments to select interface/ips may not offer big flexibility IMHO.

Imagine you want to use two ip ranges on different interfaces, but exclude some of the ips...

This will lead to horrible syntax like: --nic eth0:192.168.1.1-250:!10 --nic eth1:10.10.1.35-2.250,!1.50 ...

What if you want to do more complex things: Add weight to some IPs ? use only even/odd ones / use wildcard in interface names ? Or shrink IP range depending on current stage or elapsed time ? Use a specific IP for a given URL ... etc.

No chance we can cover all use cases from global configuration !

So... Why not doing it from javascript with a new module eg: k6/net that would provide a minimal toolbox for networking stuff , like this :

• string[] net.GetIfaces()
• string[] net.GetIfaceAddresses(Iface string)
• net.SetDefaultAddress(Iface,Address string)

This way, you can do whatever you want... from your .js.

Maybe this module could be extended to change TCP/Dialer/Resolver configuration, too.

Waiting for your comments ..

Thanks Olivier

[na--]

Thanks for also tacking this @ofauchon! I'm not sure what the best way to implement and expose this feature is and I'll share some of my concerns below.

Configuring the used NICs/IP addresses (and any advanced features like weights, IP per url, etc.) via new functions in k6 goes goes against the grain a bit - the current methods for configuring things are the .json config file, the exported options object, environment variables, and CLI flags (in that order). At the same time, like you mentioned, I also don't see how we can use the current configuration options effectively to express these types of very dynamic and complex things, so doing it via JS functions could give us better UX and more flexibility... I'm just raising the point since I don't know all of the implications of introducing these sort of new config dynamics.

For example, I'm not sure if those types of functions would be problematic for clustered/distributed execution or not. If they have to return the network interfaces for all machines in the cluster, it would add synchronization overhead and new dependencies and could potentially lead to one machine wanting to use the interface of another. But if they only return the individual network interfaces and addresses of each machine in the cluster, it could probably be a bit confusing since they would return different things in different k6 instances. And also one k6 instance won't know the details of the whole cluster, which could hinder some types of calculations regarding how many and which addresses should be used.

Also, should those functions be accessible only from the init context or should some or all of them also be usable from the default function. If so, how does that affect performance and what happens with keep-alive requests when switching IPs/NICs between iterations of a VU or in the in the middle of an iteration.

Another potential concern is the cost-benefit analysis regarding complexity and where to draw the line on a new feature set like this. Since k6 is primarily a load-testing tool, the main purpose of supporting multiple NICs and IPs is probably to get more throughput and multiple source IPs. So I'm not sure if the additional complexity needed to support different weights, shrink IP ranges when certain time has elapsed, etc. would be worth it.

I'm still not sure about the answers to any of those questions, I'm just raising them here to generate some discussion and ideas, so please don't get discouraged. I actually kind of like the idea of controlling that aspect of k6's execution via some API exposed to the JS runtime, the above wall of text was just me trying to semi-verbally work through all of the implications of that decision that I can think of :sweat_smile:

[ofauchon]

Hi na--.

Thanks for your feedback, this is an interesting point of view.

My first idea was to allow each machine to dynamically choose only from its local IPs, not IPs from a global cluster IP pool.

This way, there cannot be conflicts between k6 machines.

Problems will come if you run multiple instances of k6 on the same machine, this is true.. This requires JS to be very carefull of that (through a k6 instance UUID and hash/modulo to use only part of local IP range. ) => "great power comes great responsibility" as we sometimes say :-)

Maybe we should ask ourselves if this new complexity is worth the feature.

Anyway, I still believe JS should be able to control some aspects of configuration at runtime (not only from init). But I'm not sure what will be the implications on a future K6 cluster.

For the moment, I'll continue working on a simple implementation of IP spoofing through command line arguments & options so we can merge this first feature.

Olivier Fauchon

[FournierAlexandre]

Hello Olivier,

Thanks for sharing your thought on this!

I cannot speak for other use cases than mine at this point but our setup is quite simple. We have a storage solution that we need to check is performance in term of RPS but also bandwidth. The goal here is to ensure to serve a maximum bandwidth from a standard setup (i.e. 30 Gpbs of throughput). The goal here was only to leverage a maximum of Network power to break the system (hopefully not :))

I think maybe developing strategies could be very helpful from a starting point of view. Having on the command line a simple switch like you did '-n' and add maybe --requestDistribution RoundRobin or something similar.

But having the NIC accessible through the script will give a great deal of flexibility.

Thanks for your time!

Alex

[FournierAlexandre]

Hello Olivier,

Unfortunately, the Team won't be able to test it soon. I can do a POC on our infrastructure, is that enough on your side in term of feedback?

[ofauchon]

Hi Alexandre,

I just made a pull request so the patch is merged in the project. Howerver, all tests are welcome to see if the feature is perfectly working on many setups.

Please let us know if you like it and if it works as expected.

Olivier Fauchon

[FournierAlexandre]

Hello,

So we have integrated the fixed and test it on our Windows environment with multiple NIC and it is working as expected.

We did not see any regression from current version !

At the moment the fix make you choose a single NIC, is there a plan to use multple NIC at a time?

Thanks!

Alex