Error reporting lacking

[xapple]

I tried playdoh.map and made a small error in the definition of my test function. I was able to quickly find my mistake but in a case where the traceback would be more complex anything written using playdoh would be impossible to debug. Here is the traceback produced, there is no mention of something like "takes exactly 2 arguments (1 given)":

Python 2.7.3 (default, Sep  6 2012, 15:07:24)
Type "copyright", "credits" or "license" for more information.

IPython 0.13 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: def compute_fib(self, n):
   ...:         phi = (1 + 5**0.5) / 2
   ...:         return int(round((phi**n - (1-phi)**n) / 5**0.5))
   ...:

In [2]: import playdoh

In [3]: print playdoh.map(compute_fib, range(5), cpu=5)
Using 5 CPU(s) on the local machine
Connection error (1/5):
Connection error (2/5):
Connection error (3/5):
Connection error (4/5):
Connection error (5/5):
Connection error (1/5):
Connection error (2/5):
Connection error (3/5):
Connection error (4/5):
Connection error (5/5):
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-2f91f1abe37a> in <module>()
----> 1 print playdoh.map(compute_fib, range(5), cpu=5)

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/interface.pyc in map(*args, **kwds)
    158     kwds['disconnect'] = False
    159     myjobs = map_async(*args, **kwds)
--> 160     results = myjobs.get_results()
    161     GC.disconnect()
    162     return results

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/asyncjobhandler.pyc in get_results(self, ids)
    379 #        clients.disconnect()
    380
--> 381         results = self.concatenate(results)
    382         if self.local:
    383             close_servers(self.get_machines())

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/asyncjobhandler.pyc in concatenate(self, lists)
    345     def concatenate(self, lists):
    346         lists2 = []
--> 347         [lists2.extend(l) for l in lists]
    348         return lists2
    349

TypeError: 'NoneType' object is not iterable

In [4]:

Here is a real world example, where apparently something wasn't pickleable and made everything crash. But the user has no idea what caused this and the exception reported is uninformative.

Python 2.7.3 (default, Sep  6 2012, 15:07:24)
Type "copyright", "credits" or "license" for more information.

IPython 0.13 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from illumitag import projects; p = projects['evaluation'];  p.load(); p.count
Using 5 CPU(s) on the local machine
client: connection lost while sending the procedure,                      connecting again...
Connection error (1/5):
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/bubo/home/h3/lucass/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/bubo/home/h3/lucass/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/baserpc.py", line 366, in _execute
    self.results[index] = self.clients[index].execute(procedure)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/baserpc.py", line 275, in execute
    self.conn.send(procedure)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/connection.py", line 33, in send
    s = cPickle.dumps(obj, -1)
PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed

Traceback (most recent call last):
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/rpc.py", line 352, in process
    **procedure.kwds)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/asyncjobhandler.py", line 260, in get_results
    raise Exception("Please specify job identifiers.")
Exception: Please specify job identifiers.

Connection error (1/5):
Connection error (2/5):
Connection error (2/5):
Connection error (3/5):
Connection error (3/5):
Connection error (4/5):
Connection error (4/5):
Connection error (5/5):
Connection error (5/5):
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/bubo/home/h3/lucass/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/bubo/home/h3/lucass/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/baserpc.py", line 138, in serve
    result = self.process(client, procedure)
  File "/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/rpc.py", line 332, in process
    if procedure.is_native():
AttributeError: 'NoneType' object has no attribute 'is_native'

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-0f56b896d901> in <module>()
----> 1 from illumitag import projects; p = projects['evaluation'];  p.load(); p.count

/bubo/home/h3/lucass/repos/illumitag/illumitag/groups/aggregate.pyc in count(self)
     60     def count(self):
     61         #return sum(map(lambda p: p.count, self.pools))
---> 62         return sum(playdoh.map(lambda p: p.count, self.pools, cpu=len(self)))
     63
     64     @property

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/interface.pyc in map(*args, **kwds)
    158     kwds['disconnect'] = False
    159     myjobs = map_async(*args, **kwds)
--> 160     results = myjobs.get_results()
    161     GC.disconnect()
    162     return results

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/asyncjobhandler.pyc in get_results(self, ids)
    379 #        clients.disconnect()
    380
--> 381         results = self.concatenate(results)
    382         if self.local:
    383             close_servers(self.get_machines())

/bubo/home/h3/lucass/.local/lib/python2.7/site-packages/playdoh-0.3.1-py2.7.egg/playdoh/asyncjobhandler.pyc in concatenate(self, lists)
    345     def concatenate(self, lists):
    346         lists2 = []
--> 347         [lists2.extend(l) for l in lists]
    348         return lists2
    349

TypeError: 'DistantException' object is not iterable

In [2]:

I skimmed your publication and you seemed to have thought of lots of things, but you might have overlooked the ease-of-use ! This greatly hampers possible applications for the software...

[JohnGriffiths]

Also running into both of these kinds of errors; not sure how diagnose or fix. +1 to the issue.