set source_address for requests

[jefflowrey]

I am trying to run a number of different slave instances on a machine with more than one IP address.

I haven't been able to get the slaves to bind to anything other than the main IP address of the system.

I've spent some time digging down layers of python code looking to see where a source_address can be set. It looks like this is possible with urllib3 (at newer levels of python 2.7). But I can't figure out how to set it on the urllib3 object, or the request object, or any higher level objects that locustio uses.

This is a fairly major issue - without it, I will need to provision a lot more machines to produce the workload I need - since each IP address gets saturated at around 64k sockets. I need to test 1M+ connections.

[jefflowrey]

Hi - Has anyone from the development team looked at this issue yet?

[justiniso]

Hi @jefflowrey, I know we're a little late to the party, but help me understand your network setup. How is your machine configured with multiple IP's? Can you send the route output?

We can definitely think about exposing the source_address through the public API, but I'll have to dig into the underlying libraries and think about a sane way to do that. If you have the time or desire to take a stab yourself and open a PR, that would definitely expedite the feature.

[jefflowrey]

We've moved on to other solutions, alas. I'm not sure I quite remember where we were.

But simply imagine a machine with more than one ethernet interface, like eth0 and eth1. One is a public address, that is bound to eht0. eth1 is a private address.

I would need to make sure that any locust/locutio services were only bound and exposed to the private address, and not the public.

A simple way to do this, of course, would be to switch eht0 to be the private address and eth1 to be the publicl. But since these are cloud hosted machines, that's not really possible.

So, again, the need to ensure that locust/locution slaves and even the server are using an ip address/interface that can be configured.

[heyman]

It sounds to me like you wanted to control which IP-address Locust bound to when listening for incoming connections? If so, couldn't you firewall the machines to make sure that the Locust ports wasn't accessible from the outside network?

Or, did you need to somehow control which network interface was used for the outgoing HTTP connections? Why wouldn't the correct interface get selected automatically when making requests to an IP on the internal network?

Just trying to understand the issue :).

[jefflowrey]

Well, as I said, we've moved on... so I'm a bit fuzzy on the original question. Reading it again, it looks like I wanted to be able to set slave A to always use ip 1, slave 2 to always use ip2, and etc.

This may have been for firewall or performance reasons - I don't remember. Or even logging /performance testing issues.

[jefflowrey]

I guess in general it would be good if both the server and the clients could have ip configuration exposed to end users - rather than relying on default behavior of urrlib3 or etc.

[heyman]

Well, as I said, we've moved on... so I'm a bit fuzzy on the original question.

I totally get that and sorry for not replying back then!

Even though I guess it doesn't really matter to you now, I'll address the following just as reference for anyone else that might be interested in this in the future:

it looks like I wanted to be able to set slave A to always use ip 1, slave 2 to always use ip2, and etc.

I assume we're talking about which IP we want to send HTTP requests to. This could be done by setting the IP that we want to load test, as an environment variable when starting Locust, and then reading that environment variable in our load testing scripts. We would then start the slave processes with the correct IP set for them respectively. I guess it's kind of hackish, but I'm not sure that it's a common enough use-case to justify a feature in Locust for it.

I guess another - perhaps cleaner - way to do it could be to set a custom hostname (e.g. test-target) in /etc/hosts on the slave machines, that points to the correct IP, and then set that hostname when spawning Locust.

Again, I understand you ended up using other stuff. I'm just writing it down for future reference :).

[varianone]

I was searching for the ability to set the SRC IP for the locust process and arrived at this thread. I think what the OP meant here is that he wants to be able to bind a specific source IP address to a locust process. For example: if we have NIC eth0 with 2 IPs 192.168.1.1 and 192.168.1.2 Then, if we run 2 locust processes in slave mode, one would send the requests using .1 IP and the other using .2 IP.

The ability to generate traffic from multiple source IPs to the same destination IP is very important when testing multi node / multi core systems that rely on the IP as the hash for distributing the traffic/load. It would be great if this capability could be exposed as a command line argument when starting the process.

As a reference, wget has this capability with "--bind-address"

[gothicsquash]

Any solution yet for this issue?

Simple scenario (1 host, 2 NICs):
locust -f ./good_traffic.py --host=http://myserver:80 [--source-ip=10.10.10.1] locust -f ./bad_traffic.py --host=http://myserver:80 [--source-ip=10.10.10.2]

Is there a way to do something like that?

[griojas]

Adding to the thread.

We want to simulate users coming from different IP addresses in order to test: 1) Our load balancer sends requesters to the closest cluster (according to its IP) 1) Our HA Proxy and the sticky session mechanism works as it should keeping session data per unique IP 2) Our service behind the HA proxy can handle incoming load and stress

Note: We have only 1 target host

Is there any progress in such feature for the slaves?

[cgoldberg]

Is there any progress in such feature for the slaves?

No, but Pull Requests are welcomed

[ram-ibm]

In case someone is still looking for this feature. There are HTTPAdapters that can be leveraged for this purpose. This code shows how (works for me):

from locust import HttpLocust, TaskSet
from requests_toolbelt.adapters.source import SourceAddressAdapter
import random

def login(l):
    l.client.post("/login", {"username":"ellen_key", "password":"education"})

def logout(l):
    l.client.post("/logout", {"username":"ellen_key", "password":"education"})

def index(l):
    l.client.get("/")

def profile(l):
    l.client.get("/profile")

class UserBehavior(TaskSet):
    tasks = {index: 2, profile: 1}

    def on_start(self):
        # List of IP's that can be used as "source" - hardcoded for simplicity
        ips = ['192.168.100.10', '192.168.100.11', '192.168.100.12', '192.168.100.13']
        i = random.randint(0, len(self.ips)-1)
        self.client.mount("https://", SourceAddressAdapter(self.ips[i]))
        self.client.mount("http://", SourceAddressAdapter(self.ips[i]))
        login(self)

    def on_stop(self):
        logout(self)

class WebsiteUser(HttpLocust):
    task_set = UserBehavior
    min_wait = 5000
    max_wait = 9000

[cyberw]

Thanks for your suggested workaround @ram-ibm ! Closing this.