search

python-diamond / Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
http://diamond.readthedocs.org/
MIT License
1.74k stars 600 forks source link

windows: Can't pickle thread.lock #194

Open docwhat opened 9 years ago

docwhat commented 9 years ago

I'm getting an error about pickling a thread.lock object. I'm not sure if this is a deficiency in the default pickler (should diamond use dill instead?) or if this should never be pickled anyway.

Alternatively, something else could be going wrong that I don't understand and the thread.lock error is just hiding it. :-/

The reason I renamed my Scripts\diamond script to Scripts\diamond-cli.py is due to bug #187.

My error:

C:\>"C:\Python27\python.exe" "C:\Python27\Scripts\diamond-cli.py" --skip-pidfile --skip-change-user --skip-fork -f -l -c "C:\Program Files (x86)\diamond\conf\diamond.conf"
1432752184.65   [MainProcess:DEBUG]     Loading Handler diamond.handler.graphite.GraphiteHandler
1432752184.66   [MainProcess:DEBUG]     GraphiteHandler: Established connection to graphite server graphite.example.com:2003.
1432752184.72   [MainProcess:ERROR]     Unhandled exception: Can't pickle <type 'thread.lock'>: it's not found as thread
.lock
1432752184.72   [MainProcess:ERROR]     C:\Python27\Scripts\diamond-cli.py:300
1432752184.72   [MainProcess:ERROR]     traceback: Traceback (most recent call last):
  File "C:\Python27\Scripts\diamond-cli.py", line 292, in main
    server.run()
  File "C:\Python27\lib\site-packages\diamond\server.py", line 120, in run
    process.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 636, in _batch_appends
    save(tmp[0])
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 636, in _batch_appends
    save(tmp[0])
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock

1432752184.72   [MainProcess:ERROR]     C:\Python27\Scripts\diamond-cli.py:301
PS C:\> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError
Exception AttributeError: "'GraphiteHandler' object has no attribute 'socket'" in <bound method GraphiteHandler.__del__ of <diamond.handler.graphite.GraphiteHandler object at 0x0282B8B0>> ignored
docwhat commented 9 years ago

I tried switching to the NullHandler and it didn't help any.

mattrobenolt commented 9 years ago

Hey @docwhat, I'm sorry, but I don't run windows and have never run diamond on windows, so I'm not sure I can be of much help, but if you come up with a solution (or someone else), I'll happily review and merge.

docwhat commented 9 years ago

@mattrobenolt

I took a look at it and it is beyond me. As near as I can tell, it has something to do with the way the multiprocessing is working.

For giggles and grins, I tried adding import dill as pickle at the top, but it then had problems with a missing ._fmode attribute on ForkingPickler... so I suspect a file-handle/file-object is being pickled someplace that probably shouldn't be.

Either way, this is beyond my skills.

If you want to get windows yourself, there are VM images available from Microsoft for free for development work. You can get one of them to try stuff out.

Ciao!

themalkolm commented 8 years ago

The problem is handlers are created and then passed to the server process. Every handler has logger & lock objects and it is impossible to pickle these guys. The solution is to move handlers creation to the server process and pass around log_name instead of the actual log file.

docwhat commented 8 years ago

Thanks for explaining that. That makes sense.

themalkolm commented 8 years ago

It is quite easy to fix this. Maybe will make a patch later.

davidballano commented 8 years ago

Hi Guys, I have ran into the same problem trying to get diamond working on Windows, unfortunately this is also beyond my knowledge, thanks for taking a look.