Unexpected EOF error while evaluating objects inline

[bibsian]

Summary

I get an unexpected EOF error while evaluating objects in line.

Steps to reproduce

Inline evaluation error

y = [1, 2, 3, 4]

If I create a list like above, highlight a number within the list, and evaluate the integer with C+c C+c, I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bibsian/anaconda3/envs/fastai/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "/home/bibsian/Desktop/lvdata/lvdata/scipy/nb.py", line 13
    Y = [1
         ^
SyntaxError: unexpected EOF while parsing

My configuration

OS

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic

Result of (elpy-config)

Elpy Configuration

Virtualenv........: fastai (/home/bibsian/anaconda3/envs/fastai)
RPC Python........: 3.6.9 (/home/bibsian/anaconda3/envs/fastai/bin/python)
Interactive Python: python (/home/bibsian/anaconda3/envs/fastai/bin/python)
Emacs.............: 26.3
Elpy..............: 1.31.0
Jedi..............: 0.15.1
Rope..............: Not found (0.14.0 available)
Autopep8..........: Not found (1.4.4 available)
Yapf..............: Not found (0.28.0 available)
Black.............: Not found (19.10b0 available)
Syntax checker....: flake8 (/home/bibsian/anaconda3/envs/fastai/bin/flake8)

Elpy configuration in my init.el

;;Melpa repo

;; Added by Package.el.  This must come before configurations of
;; installed packages.  Don't delete this line.  If you don't want it,
;; just comment it out by adding a semicolon to the start of the line.
;; You may delete these explanatory comments.
(package-initialize)

(require 'package)
(add-to-list 'package-archives
             '("melpa-stable" . "https://stable.melpa.org/packages/"))

(setq ring-bell-function 'ignore)

;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Installed pacakges ;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(package-selected-packages
   (quote
    (jedi fill-column-indicator dracula-theme fold-this elpy))))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

;;;;;;;;;;;;;;;;;
;;; Enabled ;;;;;
;;;;;;;;;;;;;;;;;
(elpy-enable)

;; Allow moving between windows with Shift + (<-, ->, up, down arrows)
(windmove-default-keybindings)

(setq elpy-rpc-backend "jedi") ;; for elpy default
;;;(setq-default indent-tabs-mode nil)
(setq elpy-shell-use-project-root nil)

(when (version<= "26.0.50" emacs-version )
  (global-display-line-numbers-mode))

;;---------------------------------------------
;; this portion provide the line limit indicator
;;-----------------------------------------------
(require 'fill-column-indicator)
(define-globalized-minor-mode
 global-fci-mode fci-mode (lambda () (fci-mode 1)))
(global-fci-mode t)

(add-hook 'text-mode-hook 'turn-on-auto-fill)
(add-hook 'text-mode-hook
  '(lambda() (set-fill-column 72)))

;----------------------;
;;; Dracula dark theme;;
;----------------------;

(add-to-list 'custom-theme-load-path "~/.emacs.d/themes")
(load-theme 'dracula t)

[galaunay]

Thanks for the report (and all the details).

At the moment, Elpy doesn't support sending only the portion of a line of code. This could be implemented, but I am (genuinely) curious of what is the use case for this ?

[bibsian]

@galaunay That's very odd; I've been doing this for years when I develop on a mac (I opened the report using Ubuntu 18).

A lot my use cases are if I'm debugging a class or a function I tend to highlight an argument with a default and continue to see where some things aren't working or what they look like (I usually build out programs for data analysis/data pipelining and like to visually inspect stuff).

Here's a trivial example of what I'm use to doing on mac with a generic function; if while building out the function I want evaluate line by line to see where it's breaking I'll highlight the argument and value, evaluate them with C+c C+c, and just continue downward until I find my issue:

# I would highlight `bar=1` and `C+c C+c` to go line by line
def foo(bar=1):
    assert bar == 1, "Trivial assertion"
    # evaluate other code that utilized the argument

I get the same unexpected EOF while parsing error when trying the inline evaluation of the default argument on the Ubuntu 18 system above.

Here is my mac setup that currently allows me to do this without an EOF error as well as do what I proposed in the original post:

OS

MacBook Pro: (Retina, 15-inch, Mid 2015)
Processor: 2.8 GHz Intel Core i7
Memory: 16 GB 1600 MHz DDR3
Graphics: Intel Iris Pro 1536 MB
ProductName: Mac OS X
ProductVersion: 10.13.6 # OS High Sierra
BuildVersion: 17G65

Results of (elpy-config)

Elpy Configuration

Virtualenv........: None
RPC Python........: 3.6.8 (/anaconda3/envs/fastai/bin/python)
Interactive Python: python (/anaconda3/envs/fastai/bin/python)
Emacs.............: 26.1
Elpy..............: 1.24.0
Jedi..............: 0.15.1
Rope..............: Not found (0.14.0 available)
Autopep8..........: Not found (1.4.4 available)
Yapf..............: Not found (0.28.0 available)
Black.............: Not found (19.10b0 available)
Syntax checker....: Not found (flake8)

Elpy Configuration in my Init.el

(add-to-list 'load-path "/Users/andrew.bibian/.emacs.d/settings")
(require 'fold-this)

(eval-after-load "grep"
  '(progn
    (add-to-list 'grep-find-ignored-directories ".serverless")))

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(package-selected-packages
   (quote
    (company-jedi jedi fold-this ## moe-theme color-theme elpy)))
 '(ring-bell-function (quote ignore)))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

;; Allow moving between windows with Shift + (<-, ->, up, down arrows)
(windmove-default-keybindings)

;;
(add-hook 'yaml-mode-hook
          (lambda ()
            (define-key yaml-mode-map "\C-m" 'newline-and-indent)))

;; ----
;; Melpa repo
;; -----
(when (>= emacs-major-version 24)
  (require 'package)
  (add-to-list
   'package-archives
   '("melpa" . "http://melpa.org/packages/")
   t)
  (package-initialize))

;;--------------------------------------
;; Elpy config
;;--------------------------------------

(package-initialize)
(elpy-enable)
(setq elpy-rpc-backend "jedi") ;; for elpy default
(setq-default indent-tabs-mode nil)
(setq elpy-shell-use-project-root nil)

(when (version<= "26.0.50" emacs-version )
  (global-display-line-numbers-mode))

;;---------------------------------------------
;; this portion provide the line limit indicator
;;-----------------------------------------------
(require 'fill-column-indicator)
(define-globalized-minor-mode
 global-fci-mode fci-mode (lambda () (fci-mode 1)))
(global-fci-mode t)

(add-hook 'text-mode-hook 'turn-on-auto-fill)
(add-hook 'text-mode-hook
  '(lambda() (set-fill-column 72)))

;----------------------;
;;; Windows & Frames ;;;
;----------------------;
; language
(setq current-language-environment "English")

; don't show the startup screen
(setq inhibit-startup-screen t)
; don't show the menu bar
(menu-bar-mode nil)

; don't show the tool bar
(require 'tool-bar)
(tool-bar-mode -1)

; number of characters until the fill column
(setq fill-column 70)

; specify the fringe width for windows -- this sets both the left and
; right fringes to 10
(require 'fringe)
(fringe-mode 10)

;----------------------;
;;; Dracula dark theme;;
;----------------------;

(add-to-list 'custom-theme-load-path "~/.emacs.d/themes")
(load-theme 'dracula t)

There are systems differences for sure in terms of python version/package version but I've always been able to do this on a mac.

[galaunay]

I think this is due to commit 8130d5e1177d442bdb303f250f7e927a4b116420 that has been added in elpy 1.29 (which explains why you don't have any issue with your second configuration running elpy 1.24).

Elpy is now relying on python-shell-buffer-substring to extract the portion of code to send to the shell, and this function doesn't allow to work with portion of a line code (since this bug fix). However, It allows to do other interesting stuff, so we cannot really go back to the previous solution now.

The proper way would be to report to python.el through GNU Bug Tracker. As a temporary workaround, you could add the following modified python-shell-buffer-substring to you init.el:

(defun python-shell-buffer-substring (start end &optional nomain)
  "Send buffer substring from START to END formatted for shell.
This is a wrapper over `buffer-substring' that takes care of
different transformations for the code sent to be evaluated in
the python shell:
  1. When optional argument NOMAIN is non-nil everything under an
     \"if __name__ == \\='__main__\\='\" block will be removed.
  2. When a subregion of the buffer is sent, it takes care of
     appending extra empty lines so tracebacks are correct.
  3. When the region sent is a substring of the current buffer, a
     coding cookie is added.
  4. Wraps indented regions under an \"if True:\" block so the
     interpreter evaluates them correctly."
  (let* ((substring (buffer-substring-no-properties start end))
         (starts-at-point-min-p (save-restriction
                                  (widen)
                                  (= (point-min) start)))
         (encoding (python-info-encoding))
         (toplevel-p (zerop (save-excursion
                              (goto-char start)
                              (python-util-forward-comment 1)
                              (current-indentation))))
         (fillstr (when (not starts-at-point-min-p)
                    (concat
                     (format "# -*- coding: %s -*-\n" encoding)
                     (make-string
                      ;; Subtract 2 because of the coding cookie.
                      (- (line-number-at-pos start) 2) ?\n)))))
    (with-temp-buffer
      (python-mode)
      (when fillstr
        (insert fillstr))
      (insert substring)
      (goto-char (point-min))
      (when (not toplevel-p)
        (insert "if True:")
        (delete-region (point) (line-end-position)))
      (when nomain
        (let* ((if-name-main-start-end
                (and nomain
                     (save-excursion
                       (when (python-nav-if-name-main)
                         (cons (point)
                               (progn (python-nav-forward-sexp-safe)
                                      ;; Include ending newline
                                      (forward-line 1)
                                      (point)))))))
               ;; Oh destructuring bind, how I miss you.
               (if-name-main-start (car if-name-main-start-end))
               (if-name-main-end (cdr if-name-main-start-end))
               (fillstr (make-string
                         (- (line-number-at-pos if-name-main-end)
                            (line-number-at-pos if-name-main-start)) ?\n)))
          (when if-name-main-start-end
            (goto-char if-name-main-start)
            (delete-region if-name-main-start if-name-main-end)
            (insert fillstr))))
      ;; Ensure there's only one coding cookie in the generated string.
      (goto-char (point-min))
      (when (looking-at-p (python-rx coding-cookie))
        (forward-line 1)
        (when (looking-at-p (python-rx coding-cookie))
          (delete-region
           (line-beginning-position) (line-end-position))))
      (buffer-substring-no-properties (point-min) (point-max)))))

[bibsian]

Thanks for the information.

I added this to my init.el (at the bottom) and I can't seem to evaluate anything with C+c C+c. The buffer give me the following error: Symbol's value as variable is void: rx-constituents

If I knew more about lisp I'd try to debug but unfortunately I have no idea where to begin.

I'll go ahead and open up an issue on that link you provided and paste it here when I get to it.

[galaunay]

@rgemulla found a proper solution for this (see PR #1710). It should be merged soon, so you will just have to update to the last version of Elpy.

[rgemulla]

Funny, I hadn't seen this report. I coincidentally stumbled over the exactly same issue yesterday ;)

[bibsian]

@galaunay Should I still open the bug report for python.el or did the merge for #1710 take care of it?

[galaunay]

1710 fixed the problem for Elpy.

But It would be better to report it to python.el, have it fixed, and remove the workaround in Elpy.

The code in python.el is pretty clear about the fact that they choose to send the region from the beginning of the line (instead of just the beginning of the region). So there is certainly a reason for that, I just don't understand what it is...

[rgemulla]

I agree, it would be better to have this addressed upstream.

[bibsian]

Alright, I opened a bug report with Emacs by using m-x report-emacs-bug and I linked back to this issue.

I'm still waiting for it to pop up on bug tracker (I've never opened a bug report that way and when I sent the email my browser gave me a 400 error after sending so not 100% sure it went through - I tried to send again to be sure and my emacs told me it already got sent).

I'll keep an eye out on the bug tracker to see if it pops up and drop the link in here or resend in a week or two if it doesn't show.

[galaunay]

You should received a confirmation after ~24h in your mailbox. Did you sent emails from Emacs before ?

[bibsian]

Definitely have never sent email from emacs before and I did not receive a confirmation; I'll dig into it today.

[galaunay]

Let me know, I could do it as well if it is too much of a burden to setup everything.

[bibsian]

@galaunay If you could submit it to the bug tracker I'd definitely appreciate it; I looked into it and it's a non-trivial task for me to configure my emacs for email, thanks!