C-RET gives AttributeError/NameError in emacs-29.1

[tdhock]

hi @jorgenschaefer thanks for your wonderful elpy package, which I am using to teach the students in my classes this semester about debugging in python.

Summary

I expected that C-RET can be used to eval code from python buffer to python shell, even when pdb prompt is active in python shell. I observe that in emacs 29.1 that C-RET in python buffer gives an AttributeError/NameError (variable not defined), whereas the eval works fine if I type the same code in the python shell, it works fine (eval ok with no error). Also I observe that in emacs 27.2 that C-RET in python buffer works fine (eval ok with no error).

Steps to reproduce

put the following code in a python script file, for example test_pdb.py

import pdb
def f(x):
    pdb.set_trace()
    x+1
f(2)

then execute each line using C-RET. eventually you get to the (Pdb) prompt, at which point, move point to line "x+1" and hit C-RET -> AttributeError/NameError in emacs 29.1 (works fine in emacs 27.2). then move point to Python shell buffer and type x+1 then RET -> eval works fine (in both versions of emacs).

In image below, emacs 27.2 is on left, emacs 29.1 is on right.

The output in the emacs 29.1 Python buffer on the right above is copied below for reference:

Python 3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb

>>> def f(x):
...     pdb.set_trace()
...     x+1

>>> f(2)

> c:\users\th798\test_pdb.py(4)f()
-> x+1
(Pdb) x+1 #sent via C-RET in test_pdb.py buffer

Traceback (most recent call last):
  File "c:\Users\th798\Miniconda3\envs\2023-08-deep-learning\Lib\cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Pdb' object has no attribute 'do___PYTHON_EL_eval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 19, in __PYTHON_EL_eval
  File "c:/Users/th798/test_pdb.py", line 4, in <module>
    x+1
^
NameError: name 'x' is not defined
(Pdb) x+1 #typed at this prompt
3
(Pdb) 

My configuration

OS

windows 10

Result of (elpy-config)

Elpy Configuration

Emacs.............: 29.1
Elpy..............: 1.35.0
Virtualenv........: None
Interactive Python: python 3.11.4 (c:/Users/th798/Miniconda3/envs/2023-08-deep-learning/python.exe)
RPC virtualenv....: rpc-venv (c:/Users/th798/.emacs.d/elpy/rpc-venv)
 Python...........: c:/Users/th798/Miniconda3/pythonw.exe 3.9.5 (c:/Users/th798/Miniconda3/pythonw.exe)
 Jedi.............: 0.18.1 (0.19.0 available)
 Autopep8.........: 1.6.0 (2.0.4 available)
 Yapf.............: 0.32.0 (0.40.2 available)
 Black............: Not found (23.9.1 available)
Syntax checker....: flake8.exe (c:/Users/th798/Miniconda3/Scripts/flake8.exe)

Warnings

You have not activated a virtual env. It is not mandatory but often a
good idea to work inside a virtual env. You can use `M-x
pyvenv-activate` or `M-x pyvenv-workon` to activate one.

There is a newer version of Jedi available.

[Update jedi]

There is a newer version of the autopep8 package available.

[Update autopep8]

There is a newer version of the yapf package available.

[Update yapf]

Options

`Raised' text indicates buttons; type RET or click mouse-1 on a button
to invoke its action.  Invoke [+] to expand a group, and [-] to
collapse an expanded group.  Invoke the [Group], [Face], and [Option]
buttons below to edit that item in another window.

[+]-- Group Elpy
[+]-- Group Python
[+]-- Group Virtual Environments (Pyvenv)
[+]-- Group Completion (Company)
[+]-- Group Call Signatures (ElDoc)
[+]-- Group Inline Errors (Flymake)
[+]-- Group Code folding (hideshow)
[+]-- Group Snippets (YASnippet)
[+]-- Group Directory Grep (rgrep)
[+]-- Group Search as You Type (ido)
[+]-- Group Django extension
[+]-- Group Autodoc extension

Elpy configuration in my init.el

(elpy-enable)
(setq elpy-shell-starting-directory 'current-directory)
(setq conda-anaconda-home (expand-file-name "~/miniconda3"))
(setq conda-env-home-directory conda-anaconda-home)
(setq elpy-rpc-backend "jedi")  
(setq python-shell-interpreter "python"
      python-shell-interpreter-args "-i")

[tdhock]

by the way I would like to debug/fix this, but I'm not sure where to look in the elisp code, can you please help @jorgenschaefer ? this is the code for C-RET:

(defun elpy-shell-send-statement-and-step ()
  "Send current or next statement to Python shell and step.

If the current line is part of a statement, sends this statement.
Otherwise, skips forward to the next code line and sends the
corresponding statement."
  (interactive)
  (elpy-shell--ensure-shell-running)
  (elpy-shell--nav-beginning-of-statement)
  ;; Make sure there is a statement to send
  (unless (looking-at "[[:space:]]*$")
    (unless elpy-shell-echo-input (elpy-shell--append-to-shell-output "\n"))
    (let ((beg (save-excursion (beginning-of-line) (point)))
          (end (progn (elpy-shell--nav-end-of-statement) (point))))
      (unless (eq beg end)
        (elpy-shell--flash-and-message-region beg end)
        (elpy-shell--add-to-shell-history (buffer-substring beg end))
        (elpy-shell--with-maybe-echo
         (python-shell-send-string
          (python-shell-buffer-substring beg end)))))
    (python-nav-forward-statement)))

[gopar]

Please do not ping Jorgen. He no longer works on this project (Project status is currently unmaintained).

Any who, I feel python-shell-send-string is a suspect here. Please try using python-shell-send-string and see if the problem persists. If so, then something changed in python.el core (emacs itself) and a bug needs to be filed there. (Might be worth trying the python-shell-send-* family)

If it works, then this is a problem with elpy. Please update what you find, if anyone ever decides to tackle this they will have an idea of what broke.

Note for whoever works on this: I check the Emacs change logs for 28 and 29 but I didn't see anything that stood out

[tdhock]

Thanks for your response @gopar , sorry for pinging Jorgen, I did not know that he no loner works on this project. I tried the same with emacs-28.2, and the problem happens, so this suggests the problem started happening with some change between emacs 27 and 28. I tried using M-x python-shell-send-string x+1 RET at the (Pdb) prompt, and I get the same issue. I also confirm that this problem happens when using emacs python-mode without elpy, so I will file an issue with emacs, thanks for the help!

[tdhock]

AFAICT the issue is related to the fact that python-shell-send-string is using a new method for sending code, as of emacs 28, which involves running the code through the python function below, which is defconst python-shell-eval-setup-code in python.el

def __PYTHON_EL_eval(source, filename):
    import ast, sys
    if sys.version_info[0] == 2:
        from __builtin__ import compile, eval, globals
    else:
        from builtins import compile, eval, globals
    try:
        p, e = ast.parse(source, filename), None
    except SyntaxError:
        t, v, tb = sys.exc_info()
        sys.excepthook(t, v, tb.tb_next)
        return
    if p.body and isinstance(p.body[-1], ast.Expr):
        e = p.body.pop()
    try:
        g = globals()
        exec(compile(p, filename, 'exec'), g, g)
        if e:
            return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)
    except Exception:
        t, v, tb = sys.exc_info()
        sys.excepthook(t, v, tb.tb_next)

I put that python code in /home/tdhock/__PYTHON_EL_eval.py, then loaded it into my python, and then I ran python-shell-send-string x+1 RET which gave me the following Traceback:

Traceback (most recent call last):
  File "/home/tdhock/.local/share/r-miniconda/lib/python3.7/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'Pdb' object has no attribute 'do___PYTHON_EL_eval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tdhock/__PYTHON_EL_eval.py", line 19, in __PYTHON_EL_eval
    return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)
  File "/home/tdhock/elpy_test.py", line 1, in <module>
    import pdb
NameError: name 'x' is not defined

The above suggests that on line 19 of __PYTHON_EL_eval the code eval(compile(ast.Expression(e.value), filename, 'eval'), g, g) is incorrect/buggy. g = globals() but in this context (Pdb) we want to evaluate the expression in the context of the Pdb breakpoint, not in the global environment. Not sure how to fix.

[tdhock]

hi @larsmagne and @astoff I saw that you were the last ones to edit __PYTHON_EL_eval, https://github.com/emacs-mirror/emacs/blame/master/lisp/progmodes/python.el so I was wondering if you could please take a look at this issue, and see if it is possible/easy to fix?

To reproduce in python.el (not using elpy), put the following code in a python script file, for example test_pdb.py

import pdb
def f(x):
    pdb.set_trace()
    x+1
f(2)

Then open the file in emacs, C-c C-p to start a *Python* console, C-c C-c to send buffer, type x+1 RET at the (Pdb) prompt to observe that it works, then do M-x python-shell-send-string RET x+1 RET, which gives the following output on python console:

Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
> /home/tdhock/test_pdb.py(4)f()
-> x+1
(Pdb) x+1
3
(Pdb) 
Traceback (most recent call last):
  File "/home/tdhock/miniconda3/lib/python3.10/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'Pdb' object has no attribute 'do___PYTHON_EL_eval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 19, in __PYTHON_EL_eval
  File "<string>", line 1, in <module>
NameError: name 'x' is not defined
(Pdb) 

The issue is that there is a AttributeError/NameError, whereas I expected that this should run fine and give the same result (3) as when I type x+1 RET at the (Pdb) prompt. After looking at the definition of python-shell-send-string, I think the issue is in __PYTHON_EL_eval which seems to always eval in the global environment, which is incorrect in this case (at Pdb prompt we may be inside of a function call, with other variables that are not present in global environment). This is a regression, as this works fine in emacs 27 (issue happens in emacs 28 and 29). BTW I looked at the list of outstanding emacs bugs but I did not see any mention of this issue, https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs;include=subject%3Apython Also I tried filing this issue via M-x report-emacs-bug last week, but looking at the bug-gnu-emacs archive, I do not think my message went through https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-09/index.html

[astoff]

That's right, I'm afraid the only way to interact with the debugger now is to type directly in the shell window. On the bright side, you can now use C-RET to evaluate a complex statement and see the result of the last line (if that was an expression), in Jupyter style.

Fixing this would probably require parsing prompt strings, which, while possible, wouldn't count as "easy to fix"...

[tdhock]

hi @astoff thanks for your response. Should I report this bug elsewhere, in order to get it fixed faster? So this change was required in order to support the new feature about evaluating a complex statement and seeing the result of the last line? Can you please share an example of how that works? Has this change been documented anywhere? I looked at the NEWS.28 file from emacs, and I see only the following, which does not mention this change:

** Python mode

*** New user option 'python-forward-sexp-function'.
This allows the user easier customization of whether to use block-based
navigation or not.

*** 'python-shell-interpreter' now defaults to python3 on systems with python3.

*** 'C-c C-r' can now be used on arbitrary regions.
The command previously extended the start of the region to the start
of the line, but will now actually send the marked region, as
documented.

is there any way to configure emacs to use the previous (emacs 27) behavior, where this was working? something like (setq python-shell-send-string-method 'old) ? Thanks again very much.