Elpy seems partially incompatible with Emacs 25's 'native completion' feature

[cpitclaudel]

In Emacs 25, when starting Python with C-c C-c, I get the following message:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "ipython3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

On the other hand, if I disable elpy, start the repl, and re-enable it, everything works fine. Is elpy introducing a process filter for the python process, or something similar? That warning comes from this bit of python.el:

(defun python-shell-completion-native-setup ()
  "Try to setup native completion, return non-nil on success."

which deep down relies on explicitly doing an accept-process-output, which sounds rather bad.

In any case, I'm enclined to think that python.el is doing something wrong and elpy doing something right, but I'd love to get a bit of help pinpointing the bad interaction before opening an Emacs ticket, if we determine this to be the right course of action; especially since Emacs 25 is in a pretty deep freeze, so there's little chance that this will get fixed on the Emacs side before the release.

Let me know if I can help :)

[jorgenschaefer]

Hello, and thanks for the report. What's the value of python-shell-interpreter with and without Elpy enabled? Are you using a virtualenv in either case?

[cpitclaudel]

No virtualenv in any case. The bug happens regardless of the value I give to python-shell-interpreter in my .emacs (python3 or ipython3). Enabling or disabling elpy doesn't seem to change that value.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

Sorry, I don't think I explained this clearly. There is no error; the python repl buffer contains this after the warning is shown:

Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 
In [1]: python.el: native completion setup loaded

The reason the warning is shown seems to be that the accept-process-output call that python.el does times out without returning data; strangely enough, though, it doesn't happen reliably...

[jorgenschaefer]

There's nothing Elpy does here that should interfere with that code. I have no idea what's going on there. :-(

[cpitclaudel]

I'll close this then, and I'll try to investigate it myself next time it pops up :)

[rthoma]

I receive the same warning as cpitclaudel running Emacs 25.1.50 (9.0) built with homebrew on my mac.

[ghost]

I also have been getting this same warning, I ended up just setting python-shell-completion-native to nil, as I just couldn't get it to work.

I do have working tab-completion in python shell buffers with that setting with ipython as my interpreter though.

[cpitclaudel]

Looks like this is happening again on master :/

[cpitclaudel]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

[jorgenschaefer]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

Not much. This is a problem with python-mode in Emacs, not Elpy specifically.

I suspect this is yet another problem with the newer ipython stuff. As a wild shot, could you add (setenv "IPY_TEST_SIMPLE_PROMPT" "1") and/or (setenv "JUPYTER_CONSOLE_TEST" "1") to your config file and see if that fixes this problem?

[cpitclaudel]

Hmm. I'm not using IPython (I use the regular CPython); besides, everything works fine in both emacs -Q and in my regular Emacs, as long as I don't load Elpy.

[jorgenschaefer]

But the output you showed above was from ipython?

[cpitclaudel]

Indeed; that's what I guess I was testing out at the time. This happens with 2.7 and 3.5 as well.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

@jorgenschaefer I'm confused. Didn't you ask me to do this before? :)

I made the requested change, but there is no error to see:

Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> 

The except path is never taken.

[jorgenschaefer]

Yes I did, but that was with ipython which might have had different code paths :-) When trying to debug some weird behavior, it's confusing when the environment changes randomly.

What does M-: (python-shell-completion-native-try) return for you?

I'm still confused why anything that Elpy does would affect this. If you add (setq elpy-modules nil) before elpy-enable, does this still happen?

[cpitclaudel]

I'll try this. I'll also try to debug this on my own a bit, because this is getting complex and annoying, and I don't think I'm being very helpful. Let me create a minimal test case, and see if that helps me; otherwise, I'll at least have stuff to report here :)

[ghost]

I too am facing same problem as @cpitclaudel. When I do M-: (python-shell-completion-native-try), I get error

Debugger entered--Lisp error: (wrong-type-argument processp nil)
  process-buffer(nil)
  python-shell-completion-native-get-completions(nil nil "")
  python-shell-completion-native-try()
  eval((python-shell-completion-native-try) nil)
  eval-expression((python-shell-completion-native-try) nil)
  funcall-interactively(eval-expression (python-shell-completion-native-try) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Even with (setq elpy-modules nil) I still get the "Native completions have been disabled locally" warning and completions do not work. Seems to be due to: bug#22897

[ghost]

I have a question. How can I elpy-shell-send-region-or-buffer to an external Python interpreter process (not running in Emacs), but attached to a terminal emulator?

[cpitclaudel]

Closing this: after a while I ended up opening an Emacs bug report, and was pointed to another, existing one. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24401

Sorry for the confusion, and thanks for the help and the hard work on Elpy!

[jorgenschaefer]

Thank you for finding that upstream bug! That solves one of my headaches here :-D

[chase-dwelle]

I'm having this issue as well. What's the best way to suppress this error until the fix is pushed out through emacs?

[ChillarAnand]

You can try this

(setq python-shell-prompt-detect-failure-warning nil)

[cpitclaudel]

I pushed a fix to emacs-25, so the problem will go away in the next Emacs release :) In the meantime, you can redefine the affected function in your .emacs, as shown in https://github.com/emacs-mirror/emacs/commit/dbb341022870ecad4c9177485a6770a355633cc0

[jorgenschaefer]

Is it possible to reliably detect the problem? I would not mind having the fix in elpy.el's "backwards compatibility" section …

[cpitclaudel]

I'm not sure about reliably detecting it; but since the issue affects only one version I think it would work to just redefine the affected function in that version.

[chaturv3di]

I just stumbled upon this solution to the problem that I'm facing with Emacs25.1.1 (x86_64-w64-mingw32) on Windows7. I added the function definition in my .emacs.d/init.el file, but I continue to get the warning and the accompanying error in the Python interpreter.

It seems to me that python.el is being loaded after my emacs profile, hence the problem persists. I'm attaching a screenshot for reference. Am I doing something wrong?

[cpitclaudel]

You could wrap this redefinition in (with-eval-after-load 'python …)

[chaturv3di]

Thanks a lot. This worked. I'm attaching the screenshot of the code for whoever might be interested in it.

[jmromer]

(defun python-shell-completion-native-try ()
  "Return non-nil if can trigger native completion."
  (with-eval-after-load 'python
    '(let ((python-shell-completion-native-enable t)
           (python-shell-completion-native-output-timeout
            python-shell-completion-native-try-output-timeout))
       (python-shell-completion-native-get-completions
        (get-buffer-process (current-buffer))
        nil "_"))))

[cpitclaudel]

@jkrmr This use of with-eval-after-load doesn't make much sense to me. What did you have in mind?

[jmromer]

I'll refer that to @chaturv3di -- I just transcribed the screenshot for the benefit of future readers.

This workaround didn't actually work for me but I probably have something else going on in my case.

[cpitclaudel]

I should have looked at that screenshot closer; it looks wrong too. With-eval-after-load should be around the function, not inside it. And there's no need for the quote.

[chaturv3di]

I tried:

1. Removing the quote before (let ((...
2. Wrapping the entire defun block inside (with-eval-after-load 'python ...)

I got the same error as shown in the right half of my first screenshot. That error, and the accompanying warning about native completion goes away if my init.el file contains those lines as I have shown in the second screenshot. It may be wrong, as @cpitclaudel suggests, but it works for me.

[cpitclaudel]

@chaturv3di What your code does is essentially disable that function entirely (it says: "override this function, and every time its run, register a hook that does nothing"). AFAICT, it would be the same to use (defun python-shell-completion-native-try ()). Isn't it?

[chaturv3di]

@cpitclaudel, interesting. Providing the empty definition as you suggest here gets rid of the SyntaxError that you see in the first screenshot, but I still get this warning:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

Providing that incorrect function definition somehow gets rid of/suppresses this warning message too.

[cpitclaudel]

Right. That's because the trivial, empty definition returns nil. With (defun python-shell-completion-native-try t) you'd probably get the same as the complicated definition with with-eval and the quote.

[npostavs]

For the record, the correct fix is

(with-eval-after-load 'python
  (defun python-shell-completion-native-try ()
    "Return non-nil if can trigger native completion."
    (let ((python-shell-completion-native-enable t)
          (python-shell-completion-native-output-timeout
           python-shell-completion-native-try-output-timeout))
      (python-shell-completion-native-get-completions
       (get-buffer-process (current-buffer))
       nil "_"))))

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#44

[jojojames]

That doesn't seem to work for me. Calling run-python still gets the same/similar? error.

GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, Carbon Version 157 AppKit 1265.21)

james ~ python --version
Python 2.7.10
james ~

Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''');exec(compile(__code, '''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', 'exec'));
python.el: native completion setup loaded
>>> 

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. 

(use-package python
  :ensure t
  :mode ("\\.py\\'" . python-mode)
  :interpreter ("python" . python-mode)
  :init
  (eval-after-load 'python
    (lambda ()
      (defun python-shell-completion-native-try ()
        "Return non-nil if can trigger native completion."
        (let ((python-shell-completion-native-enable t)
              (python-shell-completion-native-output-timeout
               python-shell-completion-native-try-output-timeout))
          (python-shell-completion-native-get-completions
           (get-buffer-process (current-buffer))
           nil "_"))))))

I even tried just starting emacs without any python configuration and then evaling this before (also tried after to exhaust all possibilities) loading a python file.

(eval-after-load 'python
    (lambda ()
      (defun python-shell-completion-native-try ()
        "Return non-nil if can trigger native completion."
        (let ((python-shell-completion-native-enable t)
              (python-shell-completion-native-output-timeout
               python-shell-completion-native-try-output-timeout))
          (python-shell-completion-native-get-completions
           (get-buffer-process (current-buffer))
           nil "_")))))

[npostavs]

GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, Carbon Version 157 AppKit 1265.21)

Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''');exec(compile(__code, '''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', 'exec'));

Ah, it seems there is some separate bug on macOS.

• https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#50
• http://stackoverflow.com/questions/33726268/emacs-inferior-python-shell-shows-the-send-message-with-each-python-shell-send-r
• http://emacs.stackexchange.com/questions/29552/python-send-string-echo-begins-with-printvalue-sep-end-n-file-sy

[jojojames]

@npostavs Installing homebrew python fixed it for me.

[jorgenschaefer]

Thank you all for the work here! Is this fixed version in Emacs git already?

[npostavs]

Installing homebrew python fixed it for me.

Hmm, what's the difference with that python? Does it use readline instead of libedit? (use import readline; print readline.__doc__ to see).

Thank you all for the work here! Is this fixed version in Emacs git already?

Yes, also in 25.2-rc1 and up.

[jorgenschaefer]

Yes, also in 25.2-rc1 and up.

Excellent. Elpy should now provide this fixed version if run in an Emacs version before that.

Thank you, everyone!

[jojojames]

/usr/bin/python
>>> import readline; print readline.__doc__
Importing this module enables command line editing using libedit readline.
>>>

homebrew python
>>> import readline; print readline.__doc__
Importing this module enables command line editing using GNU readline.
>>>

[mambolevis]

I built emacs-25.2-rc2 with release flags. When using elpy-shell-send-region-or-buffer the following Message appears: Shell native completion is disabled, using fallback

I also tested emacs-25.2-rc2 from Index of /gnu/emacs/pretest/windowsand the same message is shown.

[jorgenschaefer]

@npostavs – is this intentional? (I don't know if it's a problem at all, even …)

[mambolevis]

Hi, I don't know where should I describe/report this issue, do you know where is the right place? regards, Levis

[npostavs]

I built emacs-25.2-rc2 with release flags. When using elpy-shell-send-region-or-buffer the following Message appears: Shell native completion is disabled, using fallback

If you don't get the Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python" is not part of the `python-shell-completion-native-disabled-interpreters' list. Native completions have been disabled locally. I guess this is intentional.