gevent error when locust server started and browse the website

[SQZ777]

Describe the bug

When I started the locust server then I browse the website, the log is:

python -m locust
[2019-11-13 00:24:30,516] SQZ777/INFO/locust.main: Starting web monitor at *:8089
[2019-11-13 00:24:30,517] SQZ777/INFO/locust.main: Starting Locust 0.12.2
[2019-11-13 00:24:33,820] SQZ777/ERROR/stderr: <gevent._socket3.socket object, fd=956, family=23, type=1, proto=0>: Invalid HTTP method: '\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03Yç\x95ç.d\x9aÿN$¾Ò\x03>þ\\îÅî9³\t\x8e\x90\x8fe\x18J¥D\x8cú Ê*Øî\x08[\x01í\x9c\xad©tá »\x04vðìè\x15E\x96\x8cêXaõCrö\x81\x00"zz\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x00\n'
[2019-11-13 00:24:33,822] SQZ777/ERROR/stderr: <gevent._socket3.socket object, fd=948, family=23, type=1, proto=0>: Invalid HTTP method: '\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03\x9c¦d_à\x8b[xý\x9eúaþÒ&\x11m\x00hCºY8ô\x05\x13$úF\t\x98\x94 \x13Ý7J*\x9dÂ)\x9dq_ÐNê¼\x80\x8fáñ\x01·2\x11É\x1f6hKUpÎ\x00"\x1a\x1a\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x00\n'
[2019-11-13 00:24:33,851] SQZ777/ERROR/stderr: <gevent._socket3.socket object, fd=960, family=23, type=1, proto=0>: Invalid http version: '\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03)p,.ò\x8d¸\x96ñ\x91þª¸,ûÏØ@<ÆRßÏBº!Ø\x80ñûö cb\x1dLÄ\'\x0fÖG\x84gvm±G\x975$ÏêÎï7\x10\x07sØ»*i_\x17\x00"\x8a\x8a\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x00\n'

Environment settings

• OS: Windows 10
• Python version: 3.7.5
• Locust version: 0.12.2

[heyman]

Please post a minimal reproducible code example.

[SQZ777]

class MyTaskSet(TaskSet):
    @task
    def my_task(self):
        print("executing my_task")

class MyLocust(Locust):
    task_set = MyTaskSet
    min_wait = 5000
    max_wait = 15000

[heyman]

That code works perfectly fine for me (once I add imports), so it sounds like there's some issue with your environment.

[lundstrj]

I know this is an old ticket but I want to bump it anyway.

I got this error today on a code base that did not exhibit this issue on Friday last week, but does today on Monday morning.

The issue presents itself before the swarm has been launched so I do not think it is in error in my own code. I haven't changed locust version, nor any other version. This just came out of nowhere as far as I can tell.

❯ locust -f locker.py -u 1 -r 1 [2021-10-25 10:57:15,783] Riscy.local/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces) [2021-10-25 10:57:15,789] Riscy.local/INFO/locust.main: Starting Locust 1.4.3 <gevent._socket3.socket at 0x103474160 object, fd=6, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x103474280 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x1034742e0 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x1034743a0 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x103474520 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x103474340 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x1034d3ee0 object, fd=9, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x103474580 object, fd=9, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x1034747c0 object, fd=9, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x1034742e0 object, fd=9, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n"

The logged errors just keep coming at about 1 line per second until I terminate the process.

I am on Apple silicon, running MacOS 11.6 (20G165) (Big Sur) Python 3.9.2 Locust 1.4.3

[lundstrj]

Rebooting my system and then running the exact same code again made the problem go away (for whoever searches for this issue and might come across this message. Sorry I couldn't offer you a better "solution" than to reboot).

[lundstrj]

And now it is back again. The code still runs OK but the <gevent._socket3.socket at 0x105b8cdc0 object, fd=13, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" lines make the output harder to follow.

Weird issue.

[lundstrj]

For some reason I was on an old version (1.4.3) of Locust. I moved up to 2.4.1, you know, because.

I get the exact same issue with the newer version of Locust. At this point I am guessing gevent or something Apple silicone related. I don't know how I keep hitting all these rare edge cases but for some reason I do.

[lolwhitaker]

@lundstrj I just debugged this same error message with someone and it turned out they had a config error. They has set the worker's master-port setting to use the WEB_UI port (8089) instead of using the expected/default bind port (5557). Not sure why this was the error in that case, but it went away when they corrected that config issue.

[lundstrj]

@lolwhitaker thanks for reaching out. I haven't change my worker's master-port but I do have the default 8089 for the web_ui. I'll see if I can locate the setting for the worker master port and see if fiddling with it will help.

What I don't get is why this problem temporarily goes away after I reboot, only to resurface again a little bit later in the day. Searching doesn't yield much but I wonder if it could be related to certificates perhaps?

[lundstrj]

❯ locust -f locker.py -u 1 -r 1 --master-port=5557 [2021-11-01 20:10:55,007] Riscy.local/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces) [2021-11-01 20:10:55,012] Riscy.local/INFO/locust.main: Starting Locust 2.4.1 <gevent._socket3.socket at 0x107bbca00 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x107bbc8e0 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n" <gevent._socket3.socket at 0x107bbc8e0 object, fd=8, family=30, type=1, proto=0>: Invalid HTTP method: "\x00\x00z\x00\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00'\x10\x7fÿÿÿ\n"

Assuming I used that flag correctly, it didn't fix my problem =/

[lundstrj]

@heyman Sorry to bother you but maybe we could reopen this and try to work together to find a fix? I'd be happy to try things and provide data if it could help in anyway?

[cyberw]

See if you can reproduce it with just gevent + requests (it is unlikely that it is caused by locust itself). Maybe they have seen similar issues..

As a workaround, you can also try using FastHttpUser instead and see if it works better.

[cyberw]

I’ll re-close this soon if there are no more details.

[lundstrj]

@cyberw Hello, I'm sorry, I missed that you reopened this (no notification for some reason).

I was using FastHttpUser already. I could have tried to switch to the regular HttpUser and see if that has an effect.

I have done a fair bit of searching for gevent issues that look similar to this one and the closest I could find (naturally I've lost the link) hinted that it could be a certificate related issue (Python does not use the system certificates but instead has it's own certificate handling).

I tried any and all suggested solutions related to running straight up requests (without gevent) without having to set verify=False to make the ssl errors/warning to go away (unsuccessfully I might add).

Then one day these gevent._socket3.socket warnings just stopped. Perhaps my attempts at fixing my certificate issue with certifi kicked in after a reboot or something. Perhaps a pip install --upgrade locust pulled down a newer versions of gevent (or some other library) which resolved this.

Unfortunately I don't know what I did (if anything) that caused this to stop occurring (but stop it did) and that does bother me a bit.

If this issue rears it's head again, I can attempt to make a minimal gevent + requests implementation to try to get the same issue (I do try to give back to the communities around the tools I use, it is the least I can do).

To whoever arrived here, having this same issue, hoping there would be a solution or a workaround. I suggest investigating if this could indeed be due to how Python handles ssl certificates. It is my current best guess. Maybe throw in a few reboots in there too for good measure?

[cyberw]

Thanks for the update!

[StevenBorg]

I just ran into this issue today. Basically, nothing gets sent once this error occurs on master. Here's an actual output from the logs. I get about 3 of these during startup. <gevent._socket3.socket at 0x7fbd0d1a49a0 object, fd=22, family=10, type=1, proto=0>: Invalid http version: '\x16\x03\x00\x00i\x01\x00\x00e\x03\x03U\x1c§ärandom1random2random3random4\x00\x00\x0c\x00/\x00\n'

Since I'm running in Azure Container Instances, I took the 'reboot' fix advice above and simply redeployed my infrastructure. That fixed it on the first go.

Using Locust 2.1.0, but will likely upgrade now that I'm seeing this issue. (I hadn't changed any code or my underlying container, so not sure why this popped up.)

[marcinh]

Same here - I just upgrade Locust + gevent to the latest version to avoid gevent issue, but the problem mentioned in this topic still exists - in my case there are 3 different messages.

locust-master: <gevent._socket3.socket at 0x7fdc22906ca0 object, fd=19, family=2, type=1, proto=0>: Expected GET method: '\x16\x03\x01\x010\x01\x00\x01,\x03\x03É\x89@\x81Þ\x85\x1cs«\x02\xadi%±\x9e¹Ð|\x05bÿ`\\\x1aÀlPG\x98\x04}¾\x00\x00¬À0À,À(À$À\x14À\n'

+

locust-master: <gevent._socket3.socket at 0x7f7e20206a60 object, fd=35, family=2, type=1, proto=0>: Invalid http version: '\x16\x03\x01\x010\x01\x00\x01,\x03\x03©6\x1a\x1e\x80l\x8a8Ïm½<\x1d²¹:è.\x00Ï%^9¨~~\x9c\x0fü\x9e\x8fà\x00\x00¬À0À,À(À$À\x14À\n'

+

locust-master: <gevent._socket3.socket at 0x7fd68f511e80 object, fd=95, family=2, type=1, proto=0>: Invalid HTTP method: '\x16\x03\x01\x010\x01\x00\x01,\x03\x03\xad\x80Bµ\x12Á\x9cRþ¬³ZðâÀí«êrl\x1a\x83\x89Î9#·\x9eð\x13Á´\x00\x00¬À0À,À(À$À\x14À\n'

+ there is something like this after the last error:

|ERROR  |locust.runners           |  RPCError found when receiving from client: ZMQ interrupted message

[wangonya]

I've also faced this issue. The problem on my side was that I was accessing the server using https (https:0.0.0.0:8089) instead of http (http:0.0.0.0:8089). Silly mistake but nothing in the error indicates that this might be the problem so I ended up wasting a lot of time. This issue comes up when searching for the error so I'm just leaving this here in case anyone else makes the same mistake.

[amoiz3567]

### NOO way THIS was the problem all along thnx @wangonya appretiate it so much.