search

etactica / mqtt-malaria

Attacking MQTT systems with Mosquittos (scalability and load testing utilities for MQTT environments)
BSD 2-Clause "Simplified" License
267 stars 57 forks source link

Problem on fuse.LoggingMixIn #20

Closed ppatierno closed 10 years ago

ppatierno commented 10 years ago

Hello, I have just git cloned mqtt malaria but when I lunch :

./malaria publish -h

just for help, I receive the following error :

Traceback (most recent call last): File "./malaria", line 30, in import beem.main File "/home/ppatierno/mqtt/mqtt-malaria/beem/main.py", line 34, in import beem.cmds File "/home/ppatierno/mqtt/mqtt-malaria/beem/cmds/init.py", line 2, in import beem.cmds.subscribe File "/home/ppatierno/mqtt/mqtt-malaria/beem/cmds/subscribe.py", line 34, in import beem.listen File "/home/ppatierno/mqtt/mqtt-malaria/beem/listen.py", line 166, in class MalariaWatcherStatsFS(fuse.LoggingMixIn, fuse.Operations): AttributeError: 'module' object has no attribute 'LoggingMixIn'

I have Ubuntu 12.04 and python-fuse seems to be updated to 0.2 version.

Paolo.

karlp commented 10 years ago

what does "pip freeze" say? Are you using a virtual environment for this?

This is what I've got on my fedora 19 box...

$ pip freeze | grep fuse
fusepy==2.0.2  # this is in my virtual environment...

I've also got fuse-python 0.2.1 installed from the system repositories, but I don't think it's being used.

If you do "pip install -e ." it should install from the setup.py file, which lists fusepy explicitly.

ppatierno commented 10 years ago

I'm using a virtual machine with VMWare. pip freeze tells me following...

fuse-python==0.2.1 llfuse==0.37.1

"pip install -e ." give me following...

Obtaining file:///home/ppatierno/mqtt/mqtt-malaria Running setup.py egg_info for package from file:///home/ppatierno/mqtt/mqtt-malaria

Downloading/unpacking mosquitto>=1.0 (from mqtt-malaria==0.1-75-gb242446) Running setup.py egg_info for package mosquitto

Downloading/unpacking fusepy (from mqtt-malaria==0.1-75-gb242446) Running setup.py egg_info for package fusepy

Installing collected packages: mosquitto, fusepy, mqtt-malaria Found existing installation: mosquitto 0.15 Uninstalling mosquitto: Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 228, in run requirement_set.install(install_options, global_options) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1089, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 476, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1391, in remove renames(path, new_path) File "/usr/lib/python2.7/dist-packages/pip/util.py", line 248, in renames shutil.move(old, new) File "/usr/lib/python2.7/shutil.py", line 300, in move os.unlink(src) OSError: [Errno 13] Permission denied: '/usr/share/pyshared/mosquitto-0.15.egg-info'

Storing complete log in /home/ppatierno/.pip/pip.log

What is the problem ?

karlp commented 10 years ago

You need a python virtualenv, whether you're running the machine virtually or not doesn't matter.

$ virtualenv my_env
$ . myenv/bin/activate
(my_env) $ pip install -e .

The problem you were getting was from trying to pip install -e the malaria project into your system as a whole, which is conflicted with the (very old) mosquitto that you have installed in ubuntu (at least, you would have had other problems too)

ppatierno commented 10 years ago

I executed virtualenv my_env all ok..

New python executable in my_env/bin/python Installing distribute..............................................................................................................................................................................................done. Installing pip...............done.

but wen I execute . myenv/bin/activate, I receive

bash: myenv/bin/activate: No such file or directory

karlp commented 10 years ago

My bad, I left out an _ character. Hopefully you can work out where.

ppatierno commented 10 years ago

Ok...but now I have :

(my_env)ppatierno@ppatierno-ubuntu:~/mqtt/mqtt-malaria$ ./malaria publish -h Traceback (most recent call last): File "./malaria", line 30, in import beem.main File "/home/ppatierno/mqtt/mqtt-malaria/beem/main.py", line 34, in import beem.cmds File "/home/ppatierno/mqtt/mqtt-malaria/beem/cmds/init.py", line 1, in import beem.cmds.publish File "/home/ppatierno/mqtt/mqtt-malaria/beem/cmds/publish.py", line 39, in import beem.load File "/home/ppatierno/mqtt/mqtt-malaria/beem/load.py", line 36, in import mosquitto ImportError: No module named mosquitto

I have already installed mosquitto !

karlp commented 10 years ago

No, you have installed mosquitto into your system, from the ubuntu packages, which are 0.15, and totally useless :) Further, in the virtualenv, those libraries can't be seen. This is the general point of the virtualenvs, so that you can have multiple/conflicting versions of libraries for different python projects. "pip freeze" inside the virtual env will show you all the python libraries that you have access to.

You made the virtual env, and activated it, did you then do "pip install -e ." ? See also https://github.com/remakeelectric/mqtt-malaria/blob/master/README-swarm.md#get-malaria for the first three steps of installing into a virtualenv.

I'll try and update the general readme to be more clear about how this is normally done.

ppatierno commented 10 years ago

I fogot to execute "pip install -e ."...i never used virtualenv...excuse me and thanks for your patience.

I think that you can improve the installation guide ;-)

I see that using the publish command I can't specify the topic to publish, why ?

Paolo.

karlp commented 10 years ago

Changing the topic is a different issue, if you see it as an issue :) For all the testing I was doing, it really didn't matter what the topics were, and the topics are presetup to match on both the publisher and subscriber sides, as they include information about message counts in the topic names. I'm closing this for the install portion at least, but yes, the install docs could do with work.

Panchatcharam commented 7 years ago

Hi Karlp, I'm also running into the same issue as described by ppatierno, I did follow the steps as outlined by you. Presently I'm in my Python virtual environment and I run the "pip install -e .", the following are observed on the screen.

(virtualenvironment) developer@app-dev-dt0:~/work/mosq_benchMark/mqtt-malaria-master$ pip install -e . Obtaining file:///home/developer/work/mosq_benchMark/mqtt-malaria-master Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/home/developer/work/mosq_benchMark/mqtt-malaria-master/setup.py", line 13, in version=version.get_git_version(), File "version.py", line 90, in get_git_version raise ValueError("Cannot find the version number!") ValueError: Cannot find the version number!

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /home/developer/work/mosq_benchMark/mqtt-malaria-master/

Could you throw some lights on it, I would like to use malaria to validate mosquitto.

karlp commented 7 years ago

That's not the same problem at atll. the error messages are completely different. You look like you have a tarball of the source instead of a a git clone of the code. Hence, the "get_get_version" is failing.

Panchatcharam commented 7 years ago

Thanks, Karlp. I did git clone and was able to run the tests successfully.
I did run publisher and subscriber with below options,

./malaria publish -t -n 10000 -P 200 -T 0.5
./malaria subscribe -n 10000 -N 200

and observed the response after running for a while

INFO:beem.listen:beem.listr-15784:Still waiting for 583817 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583719 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583619 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583522 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583514 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583294 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583193 messages
INFO:beem.listen:beem.listr-15784:Still waiting for 583099 messages
WARNING:beem.listen:beem.listr-15784:Drop count has increased by 666
INFO:beem.listen:beem.listr-15784:Still waiting for 582994 messages
ERROR:beem.listen:beem.listr-15784:Detected drops are occuring, aborting test!
Clientid: beem.listr-15784
Total clients tracked: 200
Total messages: 1417006
Total time: 14199.68 secs
Messages per second: 99 (10.020902 ms per message)
Test aborted, unable to gather duplicate/missing stats
Flight time mean:   503.00 ms
Flight time stddev: 298.17 ms
Flight time min:    0.32 ms
Flight time max:    8659.92 ms

what's drop here and I see the subscriber is aborted after a certain count. how this is determined, please shed some lights.

Could I change topics name with Malaria on the publisher side?

And one more observation, I did provide the following parameters as an argument to malaria, ./malaria publish -q 1 -s 1024 -n 10000 -P 100 -T 0.1

On the subscriber side I did observe below,

ValueError: invalid literal for float(): 3fa9BABDc55a4BDF8277fFbbBb80DA3ec36AE6aeDCbe44CB2BC4B8dA03D424D98BFaC7CdF2FdDBDAaBdC482eB6AA009E3dDaCb0BAeC6aA01124242802D22F16ceAe2534AF3FC744100fC2AE08ebeb4E122fb0d3f0fB76e61b83F59bBa72aa4ACc35dEbA8
ERROR:beem.listen:beem.listr-20258:Failed to parse a received message. (Is the publisher sending time-tracking information with -t?)
Traceback (most recent call last):
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/listen.py", line 90, in msg_handler
    ms = MsgStatus(msg)
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/trackers.py", line 73, in __init__
    self.time_created = time.mktime(time.localtime(float(payload_segs[0])))
ValueError: invalid literal for float(): 7A12EDEAeaFB4abE7EccB674AAAb50bacD5aF085bDaBDC6faA3c0c408aD2B34bEDeB711742EdfFaAfB196ADb6fDaA414984efbb305f8ded6f3BD1bfDA45f8Fa19c5728f3Cba3dDbcbd37ffbA7Ce97174b85FcEFCC50f45cc369E4ba07BF267bceC82F18d
ERROR:beem.listen:beem.listr-20258:Failed to parse a received message. (Is the publisher sending time-tracking information with -t?)
Traceback (most recent call last):
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/listen.py", line 90, in msg_handler
    ms = MsgStatus(msg)
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/trackers.py", line 73, in __init__
    self.time_created = time.mktime(time.localtime(float(payload_segs[0])))
ValueError: invalid literal for float(): 0939088DeB67cD928C18FfB7E3caf2161Ee0E90A5964D79E9cC8CeDaAeAb4f461ACfF9c5EFbA3aBBd53d4Cc0C3B03d5ED4DbB22ecD2B19f867FdcDAC1cDCb84a4a7bD9c2AbB0e302687fc9a7BacE1b7f8BF29f4B5853eeBFFF2eAe47AC35EB4598E57aE5
ERROR:beem.listen:beem.listr-20258:Failed to parse a received message. (Is the publisher sending time-tracking information with -t?)
Traceback (most recent call last):
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/listen.py", line 90, in msg_handler
    ms = MsgStatus(msg)
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/trackers.py", line 73, in __init__
    self.time_created = time.mktime(time.localtime(float(payload_segs[0])))
ValueError: invalid literal for float(): 68Fb4690AfdEC3B396b6ed30eD0Bf9C1ff7dA6B7b2FafdB85BDfbC44B0C1CA2fdbc6e4D1aCbdCfca76BDeaA5AfC31Ac8FAE05A9c056EC92D0b071dEbBeC4fD0cE7adCDeF372De98cfb8e5FE2Adc538F5bd6D27411CC13Ce1ab520bc40fBc1Bc0daFbc881
ERROR:beem.listen:beem.listr-20258:Failed to parse a received message. (Is the publisher sending time-tracking information with -t?)
Traceback (most recent call last):
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/listen.py", line 90, in msg_handler
    ms = MsgStatus(msg)
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/trackers.py", line 73, in __init__
    self.time_created = time.mktime(time.localtime(float(payload_segs[0])))
ValueError: could not convert string to float: CC00Ebf2b1FeFc12c6cc07F58619A1406dad133a54DF4bfDdAffE0f0E3FcfcD58382B91925DfFeE915b3f27C9bdEBC1acFEE4cB4213F4853AcCb50948016F7DC6d9F16c0aC632A0DdB9efE9dB2aef2E67fA782eBdDFfa294C9CffE4e6CADde2eaD33BCc5
ERROR:beem.listen:beem.listr-20258:Failed to parse a received message. (Is the publisher sending time-tracking information with -t?)
Traceback (most recent call last):
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/listen.py", line 90, in msg_handler
    ms = MsgStatus(msg)
  File "/home/developer/work/mosq_benchMark/mqtt-malaria/beem/trackers.py", line 73, in __init__
    self.time_created = time.mktime(time.localtime(float(payload_segs[0])))
karlp commented 7 years ago

This is a close issue. If you're still having issues please please please make your own issue.