Complete crash on remote object inspection

[GoogleCodeExporter]

I havnt been able to reduce this problem to a simple code snippet yet, but I 
thought id throw it out anyway.

I did a lot of innocent refactoring on code that worked fine earlier, and now 
my remote python engine hangs every time I type 'troublesomeobject.' during a 
debug session. It happens on both my computers. (win7, v2430)

Troublesomeobject is of a funny type; its subclassed from a basetype using 
type(), with a bunch of properties tacked on. The the object and its properties 
work fine in the code before the breakpoint where the remote engine hangs. I am 
100% positive this crashing didnt happen before my refactoring, and the 
refactoring has been of a rather innocent nature (mostly changing names, 
splitting of some modules).

To be clear; its not just the remote engine that hangs, pyscripter goes down 
with it. If I close pythonw from the task manager, pyscripter recovers with a 
message like 'error inspecting the namespace of troublesomeobject'. At least, 
it used to do that, but it seems it has stopped doing so.

Original issue reported on code.google.com by hoogendo...@gmail.com on 7 Nov 2011 at 9:12

[GoogleCodeExporter]

It would help to have a reproducible case, even a complex one attached here.

Original comment by pyscripter on 7 Nov 2011 at 9:40

[GoogleCodeExporter]

Agreed; I havnt succeeded in isolating the problem yet, but ill let you know 
when I do (the whole code would be a bit much to plonk down here)

Original comment by hoogendo...@gmail.com on 7 Nov 2011 at 8:02

[GoogleCodeExporter]

Got it: the problem appears when I have a class with a property that returns an 
object which overloads __getattribute__.

Here is a strongly minimalized illustration of the problem (ill try narrowing 
down further when I have the time)

class base_port(object):
    def __init__(self, workflow, name):
        self.workflow = workflow
        self.name = name
class input_port(base_port):
    def __call__(self, value):
        self.workflow.set_input_value(self.name, value)
class base_group(object):
    def __init__(self, workflow):
        self.workflow = workflow
class input_group(base_group):
    def __getattribute__(self, name):
        workflow = base_group.__getattribute__(self, 'workflow')
        return input_port(workflow, name)

class workflow(object):
    def set_input_value(name, value):
        pass
    @property
    def inputs(self):
        return input_group(self)

wf = workflow()

print 'type wf. in remote debug window'

Original comment by hoogendo...@gmail.com on 7 Nov 2011 at 10:05

[GoogleCodeExporter]

As far as I can tell this is valid python code, and my python interpreter seems 
to feel the same. Probably something to do with the way __getattribute__ is 
used in the inspection code?

Original comment by hoogendo...@gmail.com on 7 Nov 2011 at 10:08

[GoogleCodeExporter]

There is a problem with your __getattribute__ method.  For instance if you 
execute the code at the bottom.

wf = workflow()
inputs = wf.inputs
print(inputs.__class__)
print(inputs.__doc__)

you get

<__main__.input_port object at 0x02BE9C90>
<__main__.input_port object at 0x02BE9C90>

instead of getting a type object and a string.  So your input_group object 
breaks the basic contract of Python objects regarding the meaning of the 
__class__, __doc__, __dict__  etc. properties.

I would replace it with something like:

class input_group(base_group):
    def __getattribute__(self, name):
        if name == "port":
            workflow = base_group.__getattribute__(self, 'workflow')
            return input_port(workflow, name)
        else:
            return base_group.__getattribute__(self, name)

In that case PyScripter works fine.

Also I would write the workflow object as follows:

class workflow(object):
    def __init__(self):
        self.__input_group = None
    def set_input_value(name, value):
        pass
    @property
    def inputs(self):
        if self.__input_group is None:
            self.__input_group = input_group(self)
        return self.__input_group

Otherwise a new input_group will be created every time the property is accessed.

I haven't looked in detail where exactly the error arises but over here  
PyScripter does not crash unless you inspect the Variables Window.  My guess is 
that the problem occurs when pyscripter tries to determine the type, docstring 
and members of wf.inputs for the reason I explained above.

If you agree I will close down the issue as "As designed".

Original comment by pyscripter on 7 Nov 2011 at 11:32

[GoogleCodeExporter]

The recreation of input_group's was by design.

Your suggested fix in the __getattribute__ method will not do, but I strictly 
dont need the magic method; I can just add the desired attributes to the class 
'statically' and that works fine in all respects.

As for the breaking of python contracts: Im not sure if I completely 
understand, but I believe I was looking for __getattr__ instead; as a general 
rule, can we state that __getatribute__ requires a filtering on attribute name 
not to break your code (and tools that might depend on it)?

Original comment by hoogendo...@gmail.com on 8 Nov 2011 at 9:02

[GoogleCodeExporter]

The expectation is the someobject.__doc__ returns a possibly empty string, 
someobject.__class__ returns the object type, someobject.__dict__ returns a 
possibly empty dict-like object etc.  Although this is not enforced, I think is 
dangerous to deviate from that.  PyScripter tries to be tolerant in 
introspecting objects and I am not quite sure yet why the remote engine crashes.

Original comment by pyscripter on 8 Nov 2011 at 9:24

[GoogleCodeExporter]

As for closing down the issue; one (perhaps obvious) suggestion for the future 
would be for more graceful failure. It is probably infeasible to try and catch 
all the funny things people are allowed to do to their python objects, but a a 
simple timeout with an 'uhm sorry no idea what you are doing here; try a little 
less magic' message around all introspection code would be a big improvement 
already.

Original comment by hoogendo...@gmail.com on 8 Nov 2011 at 10:17

[GoogleCodeExporter]

Dear all,

the problem of a crash of the remote python engine happens with a DataFrame 
object of the pandas package 

When i try the following pandas example code from 
http://pandas.sourceforge.net/dsintro.html
in the Python interpreter of PyScripter 2.4.3.0 with python 2.6.7 on win32bit 
and the remote python engine (using rpyc Version 3.2.0) and pandas 0.6.0

from pandas import *
d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']), 'two' : Series([1., 
2., 3., 4.], index=['a', 'b', 'c', 'd'])}
df = DataFrame(d)

and try to use the object inspection by typing 
df.
in order to get the available methods and attributes of the DataFrame instance, 
the python engine crashes. The same happens, when i open the "Variables"-Window 
in PyScripter after having created the df object.
Using the internal python engine of PyScripter causes no crash, but the object 
introspection does not work neither.
Using the IDLE Python Interpreter, the object inspection works (after waiting 
for 2-3 seconds), 

With instances of the Series or TimeSeries class of pandas i had no problems.

This behaviour is very similar to the problem discussed in this Issue 579.
I din't know, if i should open a new issue or add a comment to thie already 
closed issue, so i decided to add a comment here. 

I guess, that the bug is caused by the way, how the IPython completion
mechanism has been implemented in pandas.

citing from http://pandas.sourceforge.net/dsintro.html:
>> The columns are also connected to the IPython completion mechanism so they 
can be tab-completed:
>> In [5]: df.fo<TAB>
>> df.foo1  df.foo

As i don't know, if this behaviour is caused by a problem in the
pandas code, or if the remote python engine of PyScripter has a bug, i
will cross-post this bug to the pystatsmodel/pandas mailing list.

Best regards,
Max

Original comment by bohnet....@googlemail.com on 10 Dec 2011 at 1:52

[GoogleCodeExporter]

The above works fine with Pandas 0.7.

Original comment by pyscripter on 7 Feb 2012 at 1:19

[GoogleCodeExporter]

I updated to pandas 0.7, but the error still occures.
>>> import pandas
>>> pandas.__version__
'0.7.0rc1'

Did you test it with 
PyScripter 2.4.3.0 with python 2.6.7 for win32bit on a windows 7 64bit system 
and the remote python engine (using rpyc Version 3.2.0) or another system 
configuration?

Maybe the error has something to do with the bug i described in 
http://code.google.com/p/pyscripter/issues/detail?id=597

Original comment by bohnet....@googlemail.com on 8 Feb 2012 at 9:36

[GoogleCodeExporter]

>>> pandas.__version__
'0.7.0rc1'
Pyscripter version 2.4.3
Python 2.7

Rpyc is bundled inside pyscripter, so no separate installation is required.  A 
separate installation of rpyc does not affect Pyscripter.

I just issued the commands you mentioned
from pandas import *
d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']), 'two' : Series([1., 
2., 3., 4.], index=['a', 'b', 'c', 'd'])}
df = DataFrame(d)
df.

There was no problem in introspecting df.

Issue 597 only affects slices so it is not relevant in the above code.

Original comment by pyscripter on 8 Feb 2012 at 10:50

[GoogleCodeExporter]

I tried it with python 2.7, using the windows installer from 
http://pypi.python.org/packages/2.7/p/pandas/pandas-0.7.0rc1.win32-py2.7.exe#md5
=88d9ba6c3f4a0bd88273c32483a03abe
With 2.7 it worked fine for me too.
Then i checked, if i have the same versions of numpy, dateutil, and scipy 
installed, but there was no differerence.
So i installed pandas again for python 2.6, this time not using the windows 
installer

http://pypi.python.org/packages/2.6/p/pandas/pandas-0.7.0rc1.win32-py2.6.exe#md5
=5e789037c028406e75e50dc584f189dc
, but typing python setup.py install using 
http://pypi.python.org/packages/source/p/pandas/pandas-0.7.0rc1.tar.gz#md5=1e0c8
d383cc038430c3e3e7af6483f04

After the re-installation of pandas it works for python 2.6 now, too.
Thanks for your help,

Max

Original comment by bohnet....@googlemail.com on 8 Feb 2012 at 6:59

[GoogleCodeExporter]

Original comment by pyscripter on 10 Mar 2012 at 2:09

• Changed state: Invalid