Locust pure python RPC not working in distributed mode (this makes, message seem confusing)

Shuliyey commented 7 years ago

Description of issue / feature request

Locust (0.8.1 and previous) have pure Python socket RPC implementation installed when pyzmq is not installed/detected, however the Python socket RPC implementation is not working in distributed mode

OS/app information

~ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.4.6
PRETTY_NAME="Alpine Linux v3.4"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"
~ # python --version
Python 3.6.3
~ # locust --version
/usr/local/lib/python3.6/site-packages/locust/rpc/__init__.py:6: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.
  warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.")
[2017-11-11 21:44:19,464] bc270b92830e/INFO/stdout: Locust 0.8.1
[2017-11-11 21:44:19,464] bc270b92830e/INFO/stdout:

Expected behavior

Either of the below two option

output a non misleading message - indicating pyzmq need to be installed pip install pyzmq for locust to work in distributed mode

~ # locust --host=http://qq.com 
/usr/local/lib/python3.6/site-packages/locust/rpc/__init__.py:6: UserError: Error: Need zmq installed to run in distributed mode, pip install pyzmq
errors.error("Error: Need pyzmq installed to run in distributed mode, please install pyzmp - pip install pyzmq")

throws the warning, and pure python RPC implementation for distributed mode should work as expected

~ # locust --host=http://qq.com --master
/usr/local/lib/python3.6/site-packages/locust/rpc/__init__.py:6: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.
warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.")
[2017-11-11 21:46:46,832] bc270b92830e/INFO/locust.main: Starting web monitor at *:8089
[2017-11-11 21:46:46,832] bc270b92830e/INFO/locust.main: Starting Locust 0.8.1

Actual behavior

throws the pure python socket RPC message, and errors out with (module 'gevent' has no attribute 'queue')

~ # locust --host=http://qq.com --master
/usr/local/lib/python3.6/site-packages/locust/rpc/__init__.py:6: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.
warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.")
[2017-11-11 21:50:43,430] bc270b92830e/INFO/locust.main: Starting web monitor at *:8089
[2017-11-11 21:50:43,430] bc270b92830e/ERROR/stderr: Traceback (most recent call last):
[2017-11-11 21:50:43,430] bc270b92830e/ERROR/stderr: File "/usr/local/bin/locust", line 11, in <module>
[2017-11-11 21:50:43,431] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,431] bc270b92830e/ERROR/stderr: sys.exit(main())
[2017-11-11 21:50:43,431] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,431] bc270b92830e/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/main.py", line 439, in main
[2017-11-11 21:50:43,432] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,432] bc270b92830e/ERROR/stderr: runners.locust_runner = MasterLocustRunner(locust_classes, options)
[2017-11-11 21:50:43,433] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,433] bc270b92830e/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/runners.py", line 247, in __init__
[2017-11-11 21:50:43,433] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,434] bc270b92830e/ERROR/stderr: self.server = rpc.Server(self.master_bind_host, self.master_bind_port)
[2017-11-11 21:50:43,434] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,434] bc270b92830e/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/rpc/socketrpc.py", line 74, in __init__
[2017-11-11 21:50:43,435] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,435] bc270b92830e/ERROR/stderr: self.event_queue = gevent.queue.Queue()
[2017-11-11 21:50:43,435] bc270b92830e/ERROR/stderr: 
[2017-11-11 21:50:43,435] bc270b92830e/ERROR/stderr: AttributeError
[2017-11-11 21:50:43,435] bc270b92830e/ERROR/stderr: :
[2017-11-11 21:50:43,436] bc270b92830e/ERROR/stderr: module 'gevent' has no attribute 'queue'
[2017-11-11 21:50:43,436] bc270b92830e/ERROR/stderr:

Environment settings (for bug reports)

OS: Alpine Linux 3.4.6
Python version: 3.6.3
Locust version: 0.8.1

Steps to reproduce (for bug reports)

locustfile.py

from locust import HttpLocust, TaskSet

def index(l):
    l.client.get("/")

class UserBehavior(TaskSet):
    tasks = {index: 1}

class WebsiteUser(HttpLocust):
    task_set = UserBehavior

run in python:latest container

| ~ @ 192-168-1-5 [zeyuye]  
| => docker run -it --rm python bash
root@a20b29169550:/# cd
root@a20b29169550:~# pip install locustio
root@a20b29169550:~# echo 'from locust import HttpLocust, TaskSet
> 
> def index(l):
>     l.client.get("/")
> 
> class UserBehavior(TaskSet):
>     tasks = {index: 1}
> 
> class WebsiteUser(HttpLocust):
>     task_set = UserBehavior' > locustfile.py
# remove pyzmq
root@a20b29169550:~# pip uninstall pyzmq
Uninstalling pyzmq-16.0.3:
  /usr/local/lib/python3.6/site-packages/pyzmq-16.0.3.dist-info/DESCRIPTION.rst
...
  /usr/local/lib/python3.6/site-packages/zmq/utils/zmq_constants.h
Proceed (y/n)? y
  Successfully uninstalled pyzmq-16.0.3
root@a20b29169550:~# locust --host=http://qq.com --master
/usr/local/lib/python3.6/site-packages/locust/rpc/__init__.py:6: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.
  warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.")
[2017-11-11 22:15:50,104] a20b29169550/INFO/locust.main: Starting web monitor at *:8089
[2017-11-11 22:15:50,106] a20b29169550/ERROR/stderr: Traceback (most recent call last):
[2017-11-11 22:15:50,108] a20b29169550/ERROR/stderr: File "/usr/local/bin/locust", line 11, in <module>
[2017-11-11 22:15:50,110] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,112] a20b29169550/ERROR/stderr: sys.exit(main())
[2017-11-11 22:15:50,113] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,114] a20b29169550/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/main.py", line 439, in main
[2017-11-11 22:15:50,117] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,118] a20b29169550/ERROR/stderr: runners.locust_runner = MasterLocustRunner(locust_classes, options)
[2017-11-11 22:15:50,119] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,120] a20b29169550/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/runners.py", line 247, in __init__
[2017-11-11 22:15:50,121] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,122] a20b29169550/ERROR/stderr: self.server = rpc.Server(self.master_bind_host, self.master_bind_port)
[2017-11-11 22:15:50,123] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,124] a20b29169550/ERROR/stderr: File "/usr/local/lib/python3.6/site-packages/locust/rpc/socketrpc.py", line 74, in __init__
[2017-11-11 22:15:50,127] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,128] a20b29169550/ERROR/stderr: self.event_queue = gevent.queue.Queue()
[2017-11-11 22:15:50,130] a20b29169550/ERROR/stderr: 
[2017-11-11 22:15:50,134] a20b29169550/ERROR/stderr: AttributeError
[2017-11-11 22:15:50,135] a20b29169550/ERROR/stderr: :
[2017-11-11 22:15:50,137] a20b29169550/ERROR/stderr: module 'gevent' has no attribute 'queue'
[2017-11-11 22:15:50,138] a20b29169550/ERROR/stderr:

Shuliyey commented 7 years ago

cgoldberg commented 7 years ago

output a non misleading message - indicating pyzmq need to be installed

the warning seems fine to me... what is misleading about it? ... and it already reccomends you install pyzmq.

It's a missing import, as described in #302 https://github.com/locustio/locust/issues/302#issuecomment-343603023

Shuliyey commented 7 years ago

@cgoldberg , (its not misleading/confusing in nature, however) it's misleading/confusing under the circumstance when pure python rpc implementation is not working properly in distributed mode.

i propose two different expected behaviour (we can pick one, my preference is the second one)

don't use pure python rpc implementation, throws error instead of warning, when pyzmq is not installed/detected
implement pure python rpc correctly, under this circumstance we can throw a warning about pyzmp not installed/detected

I believe the module 'gevent' has no attribute 'queue' error can be fixed by adding the below line

import gevent.queue

to this file https://github.com/locustio/locust/blob/0.8/locust/rpc/socketrpc.py#L3

i also did notice the latest master branch seem to have already abandoned pure python rpc implementation, and switched to pyzmp completely https://github.com/locustio/locust/tree/master/locust/rpc, it's only the 0.8 (and earlier) branch have this

cgoldberg commented 7 years ago

also did notice the latest master branch seem to have already abandoned pure python rpc

closing

Shuliyey commented 7 years ago

@cgoldberg the 0.8 branch still have this issue, pip install locustio currently installs the 0.8 version by default. branch 0.8 is missing a import gevent.queue, i've submitted a pull request https://github.com/locustio/locust/pull/681

cgoldberg commented 7 years ago

0.8 branch still have this issue

yup... it seems the rpc code removal was merged to master after 0.8 was tagged and released.

locustio / locust