Performance issues

[lvh]

There's a very noticeable (order of magnitude a second, I'd guess) delay when typing.

I've tried the following things:

• disabling python-check.sh
• disabling flymake entirely
• removing (elpy-use-ipython)
• increasing ac delays ((setq ac-auto-show-menu 1.0) (setq ac-quick-help-delay 1.2) and also the same thing with 2.0 and 2.2 as delays)
• moving the ropeproject to be inside my package instead of the repository toplevel directory (just in case it was accidentally trying to index the whole virtualenv)

... but so far nothing helps :(

Here's a video showing the issue writing a simple example class (a WTForms validator); my keyboard typing on the audio track so you can see the sync issue:

https://dl.dropbox.com/u/38476311/elpyperformanceissue.mov

(It's AAC/H.264; please let me know if you have issues viewing it and I'll convert it to something else.)

I'm running a fully updated current-generation Macbook Air with OS X 10.8 and Emacs for OS X. Any help is very appreciated :)

[jorgenschaefer]

This is most likely due to Rope being slow. Rope is the Python library used to get completions.

To verify it is indeed rope:

1) Disable auto-complete-mode (M-x auto-complete-mode). Things should be fast now. 2) If they are fast, remove the ropemacs sources from ac-sources:

(setq ac-sources
      (delq 'ac-source-nropemacs-dot
            (delq 'ac-source-nropemacs
                  ac-sources)))

You can evaluate this in a Python buffer with M-: for example. Confirm using C-h v ac-sources that those two elements are gone from the list, and then re-enable auto-complete-mode using M-x auto-complete-mode.

You should now have some minimal completion, but not the Rope-based introspection.

3) If that is the case, it's indeed Rope's fault. You can add those two elements to the list again and try to optimize things a bit:

– Change the Rope project root (C-c C-p C-o) to a directory with fewer files to be searched – Change that project root to a fast file system (I used to have a similar problem on a network file system)

If those don't help, I'm afraid your best bet is to keep the two auto-complete sources for Rope disabled for now. I'm working on a new backend system for elpy that makes it easier for me to optimize Rope, and also to use Jedi instead which might be faster, but I do not know when that will be done.

PS. Thank you for this excellent bug report. I'd love it if all bug reports were this detailed!

[jorgenschaefer]

Actually, one more thing you can try: Display eldoc-mode (M-x eldoc-mode). If that works, let me know, please.

[lvh]

Sorry for the delay; death in the family means little time to test.

Killing autocomplete appears to resolve the issue mostly, but there's some other culprit I haven't figured out yet.

Do you use elpy on OSX or some other platform?

[jorgenschaefer]

I am using GNU/Linux here. The one thing extra you might be missing is eldoc-mode, which also calls the Rope backend.

If you feel adventurous, you can try the current git development repository of elpy. You need to add the elpy/ directory in it to your PYTHONPATH. If you get it to work, you can see if that is faster; if not, pip install jedi and M-x elpy-set-backend RET jedi RET will use the Jedi backend instead, which might be faster for you.

Careful, though. The current master branch is very experimental. :-)

[lvh]

I do have eldoc-mode. Is all of it new, or just Jedi support?

[jorgenschaefer]

"The one thing extra you might be missing" was badly phrased by me, sorry. I meant that eldoc-mode is the one thing outside of auto-complete-mode that talks with Python while typing, so if you disable it, it might get rid of all of the delays you experience.

As for the git repository, that was a proposal for fixing the whole problem. (Possibly, not sure how likely that is.) The whole backend has been rewritten. That is, ropemode, ropemacs, and pymacs are gone and replaced by a new elpy Python module.

[lvh]

Ahhh! Okay, excellent, I will give it a try.

[lvh]

I'm afraid I can't figure out how to run it from git HEAD ( jorgenschaefer/elpy@6cdee0724f0669def11dbcc0fb1a10ce6c93dfde). Here's what I did:

• remove the elpy package installed through package.el
• git clone your repository into ~/.emacs.d/elpy
• run python setup.py install in that directory
• before (elpy-enable) (which was already there), I added (add-to-list 'load-path "~/.emacs.d/elpy/") (require 'elpy) (entire init.el here: https://gist.github.com/4674027)

When I start emacs, I get the following error:

Warning (initialization): An error occurred while loading `/Users/lvh/.emacs.d/init.el':

File error: Cannot open load file, virtualenv

Sorry, I'm sure I'm doing something entirely dumb. I read the installation instructions, but they are mostly intended for installing through package.el/marmalade. I didn't find any git instructions.

[jorgenschaefer]

Elpy uses a number of further packages with the latest version, virtualenv among them. You'll have to M-x package-install those manually When you install elpy that way, it pulls those dependencies by itself. The packages are, IIRC:

virtualenv highlight-indentation find-file-in-project idomenu nose

(If this is taking too much time, I hope to release a new version within the next 3-4 weeks, so might be easier to wait. :-) I also hope to be able to get rid of a number of other bugs until then.)

[lvh]

Ah; I had many of those, but not virtualenv. I didn't realize there was an elpy package for that, and I assumed I'd have all dependencies because I had previously installed it using package.el. I can open Emacs and Python files within it without error now; I'll let you know how the performance is when I've had a chance to write some code :)

Thanks again for your tireless efforts and amazing response times!

[jorgenschaefer]

How is the performance with the new backends? I've heard good things about performance increases from another source, but I'd be curious if you are still experiencing problems?

[lvh]

Yeah, it's definitely improved a lot; even the slowdowns that used to happen with everything disabled are gone! I've used it for a few days now and I've yet to see any issues.

One issue I have is that flymake-mode and ac-mode don't appear to be activated automatically. Secondly, ISTR that back when I used flymake a long time ago, being on a highlighted line with an error would show the error; but now I can't get flymake to do that. In the screenshot below, there's a highlighted error line just above my cursor, but I can't figure out (through flymake at least) what it thinks there's wrong with it... Perhaps something else is competing for the minibuffer?

Additionally, is there a way to disable ac trying to complete with symbols/names in the buffer?

Since I'm accessing the ae name, which is axiom.errors, it should only show ItemNotFound (and a few others) -- IUsernamePassword is just something I've used in this file, it's not a name available on the module, so presumably it's not coming from jedi.

I tried to find this myself but ac-sources, ac-compiled-sources were both nil. Is that supposed to happen?

[lvh]

Oh, and I have a weird issue in the following code:

from axiom import attributes
from twisted.protocols import amp

def typeFor(attr, **kwargs):
    return _typeMap[attr.__class__](**kwargs)

_typeMap = {
    attributes.text: amp.Unicode,
    attributes.bytes: amp.String
}

For the amp module, autocomplete doesn't work until after I've typed a name in full (so, e.g., at the end, amp.Uni would get me amp.Unicode); but autocomplete works fine for attributes. I'm guessing that's more of a jedi bug than an elpy bug, but I just figured I'd report it since you're looking for feedback on the new backends :)

(All of this is with current Git HEAD.)

[lvh]

Finally, with rope gone, how do I e.g. rename a local, extract a method or variable... i.e. any of the things Rope used to do besides offer autocompletion suggestions? Rope no longer appears to be available at all; should I install it separately next to elpy if I want these features?

[jorgenschaefer]

One issue I have is that flymake-mode and ac-mode don't appear to be activated automatically.

That is weird – both should be activated by `elpy-mode' when it starts.

Secondly, ISTR that back when I used flymake a long time ago, being on a highlighted line with an error would show the error; but now I can't get flymake to do that.

The one thing I found that does that in flymake is flymake-display-err-menu-for-current-line, which I thought was highly annoying. The latest development version of elpy now has C-c C-n and C-c C-p which cycle through flymake errors and additionally show the flymake error for those lines when it jumps there.

Additionally, is there a way to disable ac trying to complete with symbols/names in the buffer?

Tricky, tricky. Rope/Jedi can not always figure out all sensible completions, and then it is handy to have other things in the buffer (or other Python buffers) to offer completions. It's difficult to find out programmatically when completions are "sensible enough". You can remove all sources from `ac-sources' except for the two elpy ones. That variable should not be nil in a Python buffer. The value for me is this:

(ac-source-elpy ac-source-elpy-dot ac-source-abbrev ac-source-dictionary ac-source-words-in-same-mode-buffers)

For the amp module, autocomplete doesn't work until after I've typed a name in full

Yes, that does sound like a Jedi bug. I suspect it hasn't loaded the module until it fully completed a symbol or something? I can't reproduce this locally (no axiom/twisted installed, and a quick virtualenv attempt failed with some gcc errors :-))

Finally, with rope gone, how do I e.g. rename a local, extract a method or variable... i.e. any of the things Rope used to do besides offer autocompletion suggestions? Rope no longer appears to be available at all; should I install it separately next to elpy if I want these features?

Yes, you'll need Pymacs, ropemacs, ropemode and rope separately. Elpy didn't do anything special there, it just enabled ropemode for Python buffers. The following code in your .emacs should do all there is to it:

(require 'pymacs)
(pymacs-load "ropemacs" "rope-")

It's good to hear that the refactoring functions were useful. I had pretty bad experience with them the few times I tried them, so I did not feel bad about removing them. I might at some point implement this in elpy's backend, but I wouldn't hold my breath for that :-D

Can I close this issue as resolved?

[lvh]

Well, the issue was performance, and that's definitely been fixed :)

I'm afraid my next questions are more generic emacs questions than related to this issue. Let's just hope the bugs eventually get ironed out of Jedi. This is definitely a lot better than the old staus quo though!

As for the refactoring thing: I mostly want something like ST2's multiple cursors for quickly replacing a local. Rope was slightly smarter than a regex or find-replace at this, but unfortunately also suffered from bad slowdowns, up to the point where in a large function I'd be about as fast as rope, but it just needed more thinking.

Thanks again for the amazing responses, and thanks for your work on elpy. I've had more time to play with the new Jedi backend now, and other than the sometimes spottiness of the autocomplete suggestions (which, in Python, is pretty much going to be a given, and quite frankly is on par with the best out there right now), it's pretty great :)

[jorgenschaefer]

For "rename symbol", try M-, (iedit-mode) while on a symbol. This will highlight this symbol and all other occurrences, and editing one of them will edit all of them. Just hit M-, again to exit this mode. You can also narrow to the current method (C-x n d for "narrow to defun"), use it on a symbol to rename it, and once done, widen again (C-x n w), so you can easily edit same-named symbols only within a method. This also works with an active region for more complex strings. It's not Python-aware, but it works very well for most situations.

If you have questions about Emacs in general, you are welcome to the IRC channel #emacs on irc.freenode.net. If I'm around there (nick forcer) I'm happy to help, too.

And thank you for the very positive feedback!

[cuonglm]

@jorgenschaefer I faced the same problem with the stable version. Here is my config:

(setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
                         ("marmalade" . "https://marmalade-repo.org/packages/")
                         ("melpa" . "https://melpa.org/packages/")
                         ("melpa-stable" . "http://melpa.milkbox.net/packages/")
                         ("elpy" . "https://jorgenschaefer.github.io/packages/")))

....

;; elpy
(use-package elpy
  :ensure t
  :init
  (setq elpy-rpc-backend "jedi")
  (with-eval-after-load 'python (elpy-enable))
  :bind ("M-," . pop-tag-mark)
  :config
  (elpy-use-ipython)
  (setq elpy-rpc-backend "jedi")
  ;; fill column indicator
  (use-package fill-column-indicator
    :ensure t
    :config
    (add-hook 'after-change-major-mode-hook 'fci-mode)
    (setq fci-rule-color "LightSlateBlue")
    (setq-default fci-rule-column 79)
    (setq fci-handle-truncate-lines t))
  (add-hook 'elpy-mode-hook
            (lambda ()
              (local-set-key (kbd "C-c M-c") 'elpy-shell-switch-to-shell)
              (setq python-indent-offset 4))))

Set elpy-rpc-backend to jedi with no luck, typing text is extremely slow.

[jorgenschaefer]

@Gnouc, thanks for the report! What's the size of the Python buffer you are editing? Also, what file system is that on (local or network)?

[cuonglm]

@jorgenschaefer It's a standalone script, 32KB in size. I did all the work in local file system, an ext4 partition on Debian Jessie 8.2.

Another thing is I use company-jedi for completion.

[jorgenschaefer]

32KB is pretty large – Emacs has to write out the file on every change to get completions etc. If you set elpy-rpc-ignored-buffer-size to a lower value, say, 10000 maybe, does that speed up editing? It will disable "good" completions, though.

[cuonglm]

@jorgenschaefer Yes, setting elpy-rpc-ignored-buffer-size to 10000 speed up typing much. But I can not do completion anymore with error elpy-company-backend async timeout with args.

But you can see my above config was set elpy backend two times. if I removed the second one:

;; elpy
(use-package elpy
  :ensure t
  :init
  (setq elpy-rpc-backend "jedi")
  (with-eval-after-load 'python (elpy-enable))
  :bind ("M-," . pop-tag-mark)
  :config
  (elpy-use-ipython)
  ;; fill column indicator
  (use-package fill-column-indicator
    :ensure t
    :config
    (add-hook 'after-change-major-mode-hook 'fci-mode)
    (setq fci-rule-color "LightSlateBlue")
    (setq-default fci-rule-column 79)
    (setq fci-handle-truncate-lines t))
  (add-hook 'elpy-mode-hook
            (lambda ()
              (local-set-key (kbd "C-c M-c") 'elpy-shell-switch-to-shell)
              (setq python-indent-offset 4))))

Then the typing is not slow as rope backend, but is not as fast as setting ignore buffer. But it's acceptable. I have no ideal why setting backend two times causing this issue.

[jorgenschaefer]

That is strange indeed – I am afraid I do not know use-package, so I have no idea what it does there.

[ChillarAnand]

@jorgenschaefer use-package just ensures that the package is installed. Code under :init will be evaluated immediately where as code under :config will be evaluated lazily and code under :bind just binds keys to corresponding mode map.

[jorgenschaefer]

@ChillarAnand, I figured that is the intention, but why would that affect the behavior in this case? Does "lazily" do something that might not be obvious on the first glimpse? I do not know.