Company didn't work in Inferior shell

[azzamsa]

Summary

Company didn't work in inferior shell. Is this related to Elpy ? Can we make it better / should this reported to company ?

Steps to reproduce

Works in buffer

Not in inferior shell

My configuration

OS

Debian GNU/Linux 9.8 (stretch) x86_64

Result of (elpy-config)

Virtualenv........: global_py3 (/home/user/.virtualenvs/global_py3)
RPC Python........: 3.5.3 (/home/user/.virtualenvs/global_py3/bin/python)
Interactive Python: python (/home/user/.virtualenvs/global_py3/bin/python)
Emacs.............: 26.1
Elpy..............: 1.28.0
Jedi..............: 0.13.2
Rope..............: 0.12.0
Autopep8..........: 1.4.3
Yapf..............: 0.26.0
Black.............: Not found (18.9b0 available)
Syntax checker....: flake8 (/home/user/.virtualenvs/global_py3/bin/flake8)

Elpy configuration in my init.el

(use-package elpy
  :delight " Ep"
  :init (with-eval-after-load 'python (elpy-enable))
  :commands elpy-enable
  :bind ((:map elpy-refactor-map
               ("f" . elpy-yapf-fix-code))
         (:map inferior-python-mode-map
               ("C-c C-l" . helm-comint-input-ring)))
  :config
  ;; Use Flycheck instead of Flymake
  (when (require 'flycheck nil t)
    (setq elpy-modules (delq 'elpy-module-flymake elpy-modules))
    (add-hook 'elpy-mode-hook 'flycheck-mode))

  (setq elpy-rpc-backend "jedi")
  (add-hook 'elpy-mode-hook (lambda ()
                              (subword-mode +1))))

(use-package pyvenv
  :after elpy
  :config
  (add-hook 'pyvenv-post-activate-hooks (lambda ()
                                          (revert-buffer t t))))

(add-hook 'inferior-python-mode-hook (lambda ()
                                       (smartparens-mode 1)))

[memeplex]

This is not expected to work, completion in inferior buffer isn't based on jedi and thus is unable to understand expressions like records[0]. If that's your only problem, please close this issue. If you have a different example not involving complex expressions but just identifiers, please report it.

[azzamsa]

other 'simple' expression work. e.g os.TAB os.path.TAB

This is not expected to work, completion in inferior buffer isn't based on jedi and thus is unable to understand expressions like records[0]

Thank you, this is news to me.

[azzamsa]

@memeplex sorry, but may I know. How could original python REPL can't do this but Ipython RELP can complete records[0] ? is Ipython use thing like jedi under the hood ?

Can I get record[0] completion in inferior buffer with below config ? or do I need to use your workaround ?

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "-i --simple-prompt")

[memeplex]

Yes, ipython uses jedi.

When running in emacs inferior mode (implemented by python.el, not elpy itself) two rather different things may happen:

1. Your REPL supports readline completion, which is not the case of ipython since they introduced prompt_toolkit, although it still is the case of vanilla python (there is also https://github.com/ipython/rlipython which is quite outdated). In this case comint mode interacts with your REPL by sending TABs and stuff in order to emulate you yourself trying to autocomplete, so you get essentially the same behavior than outside emacs. This is called "native completion" in python.el.

2. Your REPL doesn't support readline completion, as is the case for modern ipython. In this case python.el fallbacks to some completion heuristics. In the case of ipython it uses ipython's own completion machinery which is indeed based on jedi. But the prefix it's passing to ipython's completion function is quite simplistic. The "prefix parser" in python.el is unable to recognize records[0] as an expression but it can parse and pass, say, records. Maybe this parser could be extended to accept more complex expressions, but this should be reported as a feature request to the emacs bug tracker in any case, since python.el is part of emacs. But hold your horses here because python.el is mostly in low maintenance mode.

[memeplex]

Can I get record[0] completion in inferior buffer with below config ? or do I need to use your workaround ?

My workaround has nothing to do with your problem, it's just fixing a regression introduced by some recent ipython changes in its completion mechanism. You can't fix the issue without changing the simplistic parser in python.el. I believe that for basic indexing expressions a simple regular expression could do the job, maybe I'll be taking a look at that in a few weeks, I'm too busy right now.

Anyway, in emacs it's not that important to have a full fledged experience at the REPL itself since the REPL is tightly integrated to the editor and it's customary to write most code in a language major mode buffer and send chunks of this code to the inferior mode buffer, instead of directly editing complex snippets of code in the REPL. Elpy provides plenty of ways to do this.

[azzamsa]

Wow, thank you for such exhausting explanations. It all make sense now.

1. Your REPL supports readline completion, which is not the c

Thanks lot for this. seems you know a lot of Emacs internal.

2. Maybe this parser could be extended to accept more complex expressions, but this should be reported as a feature request to the emacs bug tracker in any case, since python.el is part of emacs

Yes, I believe Ipyon devs put a lot of time to achieve that nice completion in repl. when working with python in Emacs, I miss what can I do with elisp and ielm.

you can't fix the issue without changing the simplistic parser in python.el.

So this mean: even I use Ipython in inferior shell, I also can't get records[0] completions ?. Emmm, I am willing to migrate my inferior shell to Ipython simply because of completion. if this the case, now I wondering why people use Ipython if inferior shell can't give completion even with Ipython 'backend'.

it's customary to write most code in a language major mode buffer and send chunks of this code to the inferior mode buffer.

Yes, I will follow this workflow from now on. I thought if only I can achieve the same experience with the lisp thing repl.

Elpy provides plenty of ways to do this.

My bad, I only know C-c C-c. Thanks I will maximize this feature instead of fighting with inferior shell completion.

Thanks a lot your time and knowledge :)

[memeplex]

Wow, thank you for such exhausting explanations. It all make sense now.

You're welcome.

even I use Ipython in inferior shell, I also can't get records[0] completions ?

Exactly. But it's not a big deal, here are some alternatives: 1. use the full editor and send fragments to the repl, 2. do _ = records[0] then autocomplete _., 3. simply evaluate records[0] and rely on the fact that _ is set to the last evaluated expression.

why people use Ipython if inferior shell can't give completion even with Ipython 'backend'.

Because of the many other features that ipython provides (https://ipython.readthedocs.io/en/stable/interactive/index.html). But it's true that its comparative advantage is ameliorated to some extent inside emacs, since many of its goodies are already offered by emacs for vanilla python. Still, it's great for debugging, profiling and other stuff you can do with magic commands.

[azzamsa]

Exactly. But it's not a big deal, here are some alternatives: 1. use the full editor and send fragments to the repl, 2. do _ = records[0] then autocomplete _., 3. simply evaluate records[0] and rely on the fact that _ is set to the last evaluated expression.

whoa, great. thanks a lot for this tricks. I will dig into.

you can do with magic commands.

Oh alright, I think I don't need that for now.

Thanks a lot for your time :)