Debugger not working for environments with different Python versions

[rhkarls]

I think this is a spyder-kernels issue, apologies if not!

Using Spyder 6 alpha 2 and spyder-kernels 3.0.0b2, with a simple .py file with one line 1/0. Spyder is installed using mamba following the instructions on https://github.com/spyder-ide/spyder/releases

Two different ways to attempt debugging 1) Debug file (CTRL+F5) 2) Run file, then use %debug magic in console

With two different environments, spyder-env where Spyder is installed (3.10) and custom_env running Python 3.11.

For 1) Spyder crashes with TypeError: 'CommsErrorWrapper' object is not subscriptable using the custom conda environment (custom_env). It does not crash when using the same environment as Spyder (spyder-env).

Exception in comms call pdb_input:

  File "C:\Users\username\AppData\Local\mambaforge\envs\spyder-env\lib\site-packages\spyder_kernels\comms\commbase.py", line 323, in _handle_remote_call
    buffer['call_args'],

TypeError: 'CommsErrorWrapper' object is not subscriptable

For 2), running the file, then %debug in ipython console raises TypeError below both with custom environment and using the spyder-env where Spyder 6a2 is installed.

Traceback (most recent call last):

  File c:\users\username\desktop\debugtest_6a2.py:8
    1/0

ZeroDivisionError: division by zero

%debug
> c:\users\username\desktop\debugtest_6a2.py(8)<module>()
      7 
----> 8 1/0

Traceback (most recent call last):

  Cell In[2], line 1
    get_ipython().run_line_magic('debug', '')

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\interactiveshell.py:2432 in run_line_magic
    result = fn(*args, **kwargs)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\magics\execution.py:469 in debug
    self._debug_post_mortem()

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\magics\execution.py:483 in _debug_post_mortem
    self.shell.debugger(force=True)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\interactiveshell.py:1146 in debugger
    self.InteractiveTB.debugger(force=True)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\ultratb.py:1253 in debugger
    self.pdb.interaction(None, exc)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\spyder_kernels\customize\spyderpdb.py:322 in interaction
    return super(SpyderPdb, self).interaction(

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\IPython\core\debugger.py:442 in interaction
    OldPdb.interaction(self, frame, tb)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\pdb.py:426 in interaction
    self._cmdloop()

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\spyder_kernels\customize\spyderpdb.py:674 in _cmdloop
    self.cmdloop()

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\spyder_kernels\customize\spyderpdb.py:714 in cmdloop
    line = self.cmd_input(self.prompt)

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\spyder_kernels\customize\spyderpdb.py:740 in cmd_input
    prompt, state=self.get_pdb_state())

  File ~\AppData\Local\mambaforge\envs\custom_env\Lib\site-packages\spyder_kernels\customize\spyderpdb.py:864 in get_pdb_state
    pdb_index -= sum(hidden[:pdb_index])

TypeError: unsupported operand type(s) for +: 'int' and 'str'

Versions

• Spyder version: 6.0.0a2 (conda)
• Python version: 3.10.12 64-bit
• Qt version: 5.15.8
• PyQt5 version: 5.15.9
• Operating System: Windows-10-10.0.19045-SP0

Dependencies

# Mandatory:
atomicwrites >=1.2.0                                                        :  1.4.1 (OK)
chardet >=2.0.0                                                             :  5.2.0 (OK)
cloudpickle >=0.5.0                                                         :  2.2.1 (OK)
cookiecutter >=1.6.0                                                        :  2.3.0 (OK)
diff_match_patch >=20181111                                                 :  20230430 (OK)
intervaltree >=3.0.2                                                        :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0,!=8.11.0,!=8.12.0,!=8.12.1 :  8.15.0 (OK)
jedi >=0.17.2,<0.19.0                                                       :  0.18.2 (OK)
jellyfish >=0.7                                                             :  1.0.0 (OK)
jsonschema >=3.2.0                                                          :  4.19.0 (OK)
keyring >=17.0.0                                                            :  24.2.0 (OK)
nbconvert >=4.0                                                             :  7.8.0 (OK)
numpydoc >=0.6.0                                                            :  1.5.0 (OK)
paramiko >=2.4.0                                                            :  3.3.1 (OK)
parso >=0.7.0,<0.9.0                                                        :  0.8.3 (OK)
pexpect >=4.4.0                                                             :  4.8.0 (OK)
pickleshare >=0.4                                                           :  0.7.5 (OK)
psutil >=5.3                                                                :  5.9.5 (OK)
pygments >=2.0                                                              :  2.16.1 (OK)
pylint >=2.5.0,<3.0                                                         :  2.17.5 (OK)
pylint_venv >=3.0.2                                                         :  3.0.2 (OK)
pyls_spyder >=0.4.0                                                         :  0.4.0 (OK)
pylsp >=1.7.4,<1.8.0                                                        :  1.7.4 (OK)
pylsp_black >=1.2.0,<3.0.0                                                  :  1.3.0 (OK)
pyuca >=1.2                                                                 :  1.2 (OK)
qdarkstyle >=3.0.2,<3.2.0                                                   :  3.1 (OK)
qstylizer >=0.2.2                                                           :  0.2.2 (OK)
qtawesome >=1.2.1                                                           :  1.2.3 (OK)
qtconsole >=5.4.2,<5.5.0                                                    :  5.4.4 (OK)
qtpy >=2.1.0                                                                :  2.4.0 (OK)
rtree >=0.9.7                                                               :  1.0.1 (OK)
setuptools >=49.6.0                                                         :  68.2.2 (OK)
sphinx >=0.6.6                                                              :  7.2.6 (OK)
spyder_kernels >=3.0.0b2,<3.0.0b3                                           :  3.0.0b2 (OK)
textdistance >=4.2.0                                                        :  4.5.0 (OK)
three_merge >=0.1.1                                                         :  0.1.1 (OK)
watchdog >=0.10.3                                                           :  3.0.0 (OK)
zmq >=22.1.0                                                                :  25.1.1 (OK)

# Optional:
cython >=0.21                                                               :  None (NOK)
matplotlib >=3.0.0                                                          :  None (NOK)
numpy >=1.7                                                                 :  None (NOK)
pandas >=1.1.1                                                              :  None (NOK)
scipy >=0.17.0                                                              :  None (NOK)
sympy >=0.7.3                                                               :  None (NOK)

[ccordoba12]

Hey @rhkarls, thanks for reporting. I think this is a problem with IPython 8.15, so could you downgrade to 8.14 and try again?

[ccordoba12]

@impact27, could you take a look at this one? I was able to reproduce point 2) above and it only affects master.

[rhkarls]

Hey @rhkarls, thanks for reporting. I think this is a problem with IPython 8.15, so could you downgrade to 8.14 and try again?

Indeed, downgrading to 8.14 works for spyder_env.

However, for the custom env downgrading to 8.14 does not work.

In the case of 1) above: same CommsErrorWrapper as in the initial report. In the case of 2) above: now this also causes the CommsErrorWrapper, not TypeError as in the initial report with 8.15.

[impact27]

The "TypeError: unsupported operand type(s) for +: 'int' and 'str'" should be fixed by https://github.com/spyder-ide/spyder-kernels/pull/467 so point 2) should not happen in master?

[impact27]

for point 1) https://github.com/spyder-ide/spyder-kernels/pull/476 will send the correct error which is maybe a cloud pickle issue

[ccordoba12]

@rhkarls, Spyder alpha3 will be released in a couple of days, so let us know if this error persists with that version.

[rhkarls]

Thanks! And I want to add that the debugging plugin is such an amazing improvement for Spyder, great stuff!

With 6a3 it works as expected with python 3.10 using both Spyder interpreter and custom interpreter. Testing with 3.11 and 3.12 in the custom environment it does however not work as expected, as the debugging does not properly enter the interactive debugger or show the stack in the debugger plugin.

How it works with 3.10 custom env:

And using 3.12 in custom env (same for 3.11):

Exception shown in the gif:

Exception in comms call get_current_frames:
  File "C:\Users\user\AppData\Local\mambaforge\envs\spyder-6a3\lib\site-packages\spyder_kernels\comms\commbase.py", line 298, in _comm_message
    buffer = cloudpickle.loads(msg['buffers'][0])
AttributeError: 'FrameSummary' object has no attribute 'end_lineno'

[rhkarls]

In case it helps: as far as I can see this error remains on 6.0.0a5 and spyder-kernels 3.0.0b5 if Spyder is installed in an env with python 3.10.

When Spyder 6.0.0a5 is installed with Python 3.11 the examples above work as expected with Spyders own environment and with custom environments with python 3.10, 3.11 and 3.12.

[impact27]

Could you try with https://github.com/spyder-ide/spyder-kernels/pull/492 ?

[ccordoba12]

@impact27, I was able to reproduce this. And after applying your patch in #492, now I'm seeing this error:

Exception in comms call get_current_frames:
  File "/home/carlos/Projects/spyder/spyder/external-deps/spyder-kernels/spyder_kernels/comms/commbase.py", line 298, in _comm_message
    buffer = cloudpickle.loads(msg['buffers'][0])
AttributeError: Can't get attribute '_class_setstate' on <module 'cloudpickle.cloudpickle' from '/home/carlos/miniconda3/envs/py310-pip/lib/python3.10/site-packages/cloudpickle/cloudpickle.py'>

This happens after clicking in the Interrupt execution button because the debugger also hangs in this case.

[impact27]

That is not an helpful error :/ is this only an issue when you cross python versions correct? @ccordoba12 what versions os python did you use?

[ccordoba12]

That is not an helpful error :/

Sorry, that's what I'm seeing but I can get more info if you need it.

is this only an issue when you cross python versions correct?

Yep, I confirmed it is.

@ccordoba12 what versions os python did you use?

Python 3.10 in the frontend (i.e. for Spyder) and 3.12 for the kernel.

[impact27]

Sorry, that's what I'm seeing but I can get more info if you need it.

I assume _class_setstate indeed exists (https://github.com/cloudpipe/cloudpickle/blob/f111f7ab6d302e9b1e2a568d0e4c574895db6a6e/cloudpickle/cloudpickle.py#L1150) and that the AttributeError was raised in this method. Can you easily debug and check what the arguments are? (obj, state, attrname, attr)

[ccordoba12]

After making the cloudpickle versions match, I got this error:

Exception in comms call get_current_frames:
  File "/home/carlos/Projects/spyder/spyder/external-deps/spyder-kernels/spyder_kernels/comms/commbase.py", line 298, in _comm_message
    buffer = cloudpickle.loads(msg['buffers'][0])
TypeError: code expected at most 16 arguments, got 18

which a quick googling pointed to this issue: https://github.com/cloudpipe/cloudpickle/issues/451.

And this https://github.com/cloudpipe/cloudpickle/issues/451#issuecomment-1501050455 on that issue mentions that, according to Cloudpickle's Readme

Cloudpickle can only be used to send objects between the exact same version of Python.

which is still present in it.

But after taking a closer look at Cloudpickle's code, I found the _code_reduce function (thanks to https://github.com/cloudpipe/cloudpickle/pull/467):

https://github.com/cloudpipe/cloudpickle/blob/f111f7ab6d302e9b1e2a568d0e4c574895db6a6e/cloudpickle/cloudpickle.py#L813

If I understand it correctly, it means that:

• Python 3.11 Pdb serialized frames are compatible with 3.12, but I'm not sure if they are with older versions (probably not).
• Python 3.10 frames are compatible with older versions but incompatible with newer ones.
• Python 3.8 and 3.9 are compatible among themselves and with Python 3.10, but not newer ones.

I tested the second case and it's working as expected (without the changes in PR #492). I'll test the first one next week, unless @rhkarls beats me to it.

[impact27]

Cloudpickle can only be used to send objects between the exact same version of Python.

That is not ideal. I guess the problem is that if named tuple is not the same between versions of python then you can't do anything?

I tried to change the definition of SpyderFrameSummary to a custom frame but this will be a problem more generally.

[impact27]

Most comms send and receive basic objects (strings, list, dict, float) but some are more general and this will be a problem. (get/set_value for example)

[impact27]

@ccordoba12 with a few adjustments we can replace cloudpickle with json so this class of problem disappears

[ccordoba12]

Ok, I saw your PRs about it. But I'd like to know how get/set_value is going to work without Cloudpickle because I think we need it to serialize arbitrary Python objects. And I doubt serializing them with json is enough for all objects.

Is it not possible to send the frame info back to Spyder using a dict to avoid doing removing Cloudpickle?

[impact27]

The problem here is that cloudpickle only works for the exact same version of python, and we use in general different versions of python, and therefore can not use cloudpickle. I tried replacing cloudpickle with json and all the tests are passing. In general if this is a problem we just need to encode in the calling function whatever object we want to send between the kernel and frontend. But I think we need to be careful about doing that. The current solution hides this complexity and introduces hard to detect and debug bugs. Do you know of any example where this might be a problem?

[impact27]

What we could do is to use cloudpickle only in get/set_value?

[ccordoba12]

The problem here is that cloudpickle only works for the exact same version of python, and we use in general different versions of python, and therefore can not use cloudpickle.

Sure, I understand that. But we haven't had any serious issues so far (as far as I recall this is the first one).

I tried replacing cloudpickle with json and all the tests are passing.

Yeah, but we're testing simple objects (Numpy arrays and dataframes) which (I guess) are easily serializable with json.

In general if this is a problem we just need to encode in the calling function whatever object we want to send between the kernel and frontend. But I think we need to be careful about doing that. The current solution hides this complexity and introduces hard to detect and debug bugs.

Ok, that seems quite reasonable.

Do you know of any example where this might be a problem?

No, but I doubt our users wouldn't take long to find problems with this new approach because they're using it to view all kind of objects.

However, I wouldn't like to discard what you've done but complement it, if possible, by sending objects with Cloudpickle and Jsonpickle to be able to display objects created with libraries not available on the Spyder side. That would be a significant improvement for users of our installers because it's not easy to install other libraries on them.

[impact27]

I guess most users are using the same interpreter for the kernel and frontend, so the issue with cloudpickle doesn’t show up that much.

If the only use for cloudpickle is the variable explorer, then it should be used only there. This will greatly simplify the handling of errors. We could for example serialise with cloudpickle and then send the base64 representation by comms so there is no need to have more complexity in the comms

the more general question I have because I am not that much knowledgeable with the variable explorer is: how do we edit a generic python object? Don’t we need a specific editor for each class?

[impact27]

If the only use for cloudpickle is the variable explorer, then it should be used only there. This will greatly simplify the handling of errors. We could for example serialise with cloudpickle and then send the base64 representation by comms so there is no need to have more complexity in the comms

I tried that in https://github.com/spyder-ide/spyder/pull/22120

[jitseniesen]

the more general question I have because I am not that much knowledgeable with the variable explorer is: how do we edit a generic python object? Don’t we need a specific editor for each class?

We have one editor called ObjectExplorer (IIRC) which works on arbitrary objects, but it can only be used to view objects, not to edit them. We have separate editors for list/set/tuple, numpy arrays and pandas dataframe.

Actually, the use of cloudpickle in the variable explorer is also problematic,

However, I wouldn't like to discard what you've done but complement it, if possible, by sending objects with Cloudpickle and Jsonpickle to be able to display objects created with libraries not available on the Spyder side. That would be a significant improvement for users of our installers because it's not easy to install other libraries on them.

I don't understand this. How can we unpickle an object if its class is not installed on the Spyder side?

[impact27]

I don't understand this. How can we unpickle an object if its class is not installed on the Spyder side?

from cloud pickle doc:

Among other things, cloudpickle supports pickling for lambda functions along with functions and classes defined interactively in the __main__ module (for instance in a script, a shell or a Jupyter notebook).

One advantage of my approach in the PR is that it makes it easy to find a new solution for this in the future. Indeed cloud pickle is only used in get/set_value so another solution can easily be deployed without affecting the rest of the comms.

[ccordoba12]

I don't understand this. How can we unpickle an object if its class is not installed on the Spyder side?

By using Jsonpickle, which creates a Json representation of a Python object that we can display in a Json tree viewer. So, the object wouldn't be editable but at least users would be able to check its internals.

[jitseniesen]

By using Jsonpickle, which creates a Json representation of a Python object that we can display in a Json tree viewer. So, the object wouldn't be editable but at least users would be able to check its internals.

Got it, we don't unpickle but read the json directly. That sounds like a good approach.