Typing latency in the editor due to folding

mrclary commented 3 years ago

Description

What steps will reproduce the problem?

After release of 4.2.2, I noticed typing latency in the Editor. I know that we had solved performance issues with Editor in the past, but it seems to have resurfaced.

I suspect the issue was introduced with PR #13864 since I don't see the issue in artifact from PR #14719.

Illustrated in the attached gif: note the hiccups while typing "jumped over the lazy dog". That sequence was typed (relatively) smoothly, but you can see the latency followed by bursts.

sluggish_editor

Versions

Spyder version: 4.2.2
Python version: 3.9.1
Qt version: 5.12.10
PyQt5 version: 5.12.3
Operating System: Darwin 19.6.0

Dependencies


# Mandatory:
applaunchservices >=0.1.7       :  0.2.1 (OK)
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.8.0 (OK)
pylint >=1.0                    :  2.7.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.2             :  0.3.2 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
setuptools >=39.0.0             :  52.0.0 (OK)
sphinx >=0.6.6                  :  3.5.1 (OK)
spyder_kernels >=1.10.2;<1.11.0 :  1.10.2 (OK)
textdistance >=4.2.0            :  4.2.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3;<2.0.0        :  1.0.2 (OK)
zmq >=17                        :  22.0.3 (OK)

# Optional:
cython >=0.21                   :  0.29.22 (OK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.20.1 (OK)
pandas >=1.1.1                  :  1.2.2 (OK)
scipy >=0.17.0                  :  1.6.1 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

mrclary commented 3 years ago

Is there a way for us to test this more rigorously and more accurately? It seems someone had done programmatic testing when we had problems before...

It would be nice if we could include something in our unit tests that scored the performance in some way, perhaps using a large (~1000 line) file, and asserted the performance exceeded some minimum threshold.

juanis2112 commented 3 years ago

Thanks for reporting this @mrclary. I think Edgar might know more about this. @andfoy can you please take a look at this one?

ccordoba12 commented 3 years ago

PR #13864 should have helped to solve latency issues, not introduce them again. That's because we're now waiting until a certain time before requesting symbols and folding, instead of doing it right after linting results arrive from the PyLS. In any case, I can't reproduce this problem on Linux and Windows.

What happens if you hide the class and functions selector?

mrclary commented 3 years ago

I'll check it out...

mrclary commented 3 years ago

Okay, I've used a more repeatable setup and determined that it seems to manifest in large files only. I've installed the keyboard package and used the following code to reproduce the issue.

In [1]: import time, keyboard
In [2]: time.sleep(3)
   ...: keyboard.write('The quick brown fox ', delay=0.05)
   ...: time.sleep(1.5)
   ...: keyboard.write('jumped over the lazy dog.', delay=0.05)

Open Spyder and open a large file; I used spyder/spyder/app/mainwindow.py.
Run the above code in the console.
Before the 3s sleep time is elapsed, return focus to the Editor
Observe the first block of text entered smoothly
With the 1.5s intermission, some repainting in the Editor occurs
Observe the initial sluggishness then final burst in the second block of text

4.2.2

sluggish_editor

I performed the above procedure both with and without the Class/Function Selector and there was no difference.

I've attached lsp logs and a spyder log (SPYDER_DEBUG=3 Applications/Spyder.app/Contents/MacOS/Spyder &> spyder.log). There seems to me to be a glut of log messages nearly identical to the following that appear to correspond to every "keystroke", then log messages corresponding to the 1.5s intermission, after which the previous log messages resume a less frequent cycle.

keystrokes

``` 2021-02-26 18:19:09,992 [DEBUG] [spyder.plugins.editor.plugin] -> Perform request textDocument/didChange for: /Users/rclary/Documents/Python/spyder/spyder/app/mainwindow.py 2021-02-26 18:19:09,993 [DEBUG] [spyder.plugins.completion.languageserver.client] -> Perform request textDocument/didChange with id 25 2021-02-26 18:19:09,994 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 23 Got response from kite 2021-02-26 18:19:09,995 [DEBUG] [spyder.plugins.completion.fallback.actor] -> Perform request textDocument/didChange with id 23 2021-02-26 18:19:09,995 [DEBUG] [spyder.plugins.completion.snippets.actor] -> Perform request textDocument/didChange with id 23 2021-02-26 18:19:09,996 [DEBUG] [urllib3.connectionpool] -> Starting new HTTP connection (24): 127.0.0.1:46624 2021-02-26 18:19:10,050 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 21 timed out 2021-02-26 18:19:10,064 [DEBUG] [spyder.plugins.editor.plugin] -> Perform request textDocument/didChange for: /Users/rclary/Documents/Python/spyder/spyder/app/mainwindow.py 2021-02-26 18:19:10,064 [DEBUG] [spyder.plugins.completion.languageserver.client] -> Perform request textDocument/didChange with id 26 2021-02-26 18:19:10,065 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 24 Got response from kite 2021-02-26 18:19:10,066 [DEBUG] [spyder.plugins.completion.snippets.actor] -> Perform request textDocument/didChange with id 24 2021-02-26 18:19:10,067 [DEBUG] [spyder.plugins.completion.fallback.actor] -> Perform request textDocument/didChange with id 24 2021-02-26 18:19:10,068 [DEBUG] [urllib3.connectionpool] -> Starting new HTTP connection (25): 127.0.0.1:46624 2021-02-26 18:19:10,117 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 22 timed out ```

Intermission

``` 2021-02-26 18:19:11,798 [DEBUG] [spyder.plugins.completion.languageserver.client] -> python response: textDocument/publishDiagnostics 2021-02-26 18:19:12,100 [DEBUG] [spyder.plugins.editor.widgets.editor] -> Added thread to queue 2021-02-26 18:19:12,151 [DEBUG] [spyder.plugins.editor.widgets.editor] -> Updating queue: 2021-02-26 18:19:12,151 [DEBUG] [spyder.plugins.editor.widgets.editor] -> started: 0 2021-02-26 18:19:12,151 [DEBUG] [spyder.plugins.editor.widgets.editor] -> pending: 1 2021-02-26 18:19:12,152 [DEBUG] [spyder.plugins.editor.widgets.editor] -> ===>starting: 2021-02-26 18:19:12,165 [DEBUG] [spyder.plugins.editor.widgets.editor] -> Updating queue: 2021-02-26 18:19:12,165 [DEBUG] [spyder.plugins.editor.widgets.editor] -> started: 0 2021-02-26 18:19:12,165 [DEBUG] [spyder.plugins.editor.widgets.editor] -> pending: 0 2021-02-26 18:19:12,324 [DEBUG] [spyder.plugins.editor.plugin] -> Perform request textDocument/foldingRange for: /Users/rclary/Documents/Python/spyder/spyder/app/mainwindow.py 2021-02-26 18:19:12,324 [DEBUG] [spyder.plugins.completion.languageserver.client] -> Perform request textDocument/foldingRange with id 42 2021-02-26 18:19:12,325 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 40 Got response from kite 2021-02-26 18:19:12,325 [DEBUG] [spyder.plugins.completion.fallback.actor] -> Perform request textDocument/foldingRange with id 40 2021-02-26 18:19:12,325 [DEBUG] [spyder.plugins.completion.snippets.actor] -> Perform request textDocument/foldingRange with id 40 2021-02-26 18:19:12,325 [DEBUG] [spyder.plugins.editor.plugin] -> Perform request textDocument/documentSymbol for: /Users/rclary/Documents/Python/spyder/spyder/app/mainwindow.py 2021-02-26 18:19:12,326 [DEBUG] [spyder.plugins.completion.languageserver.client] -> Perform request textDocument/documentSymbol with id 43 2021-02-26 18:19:12,326 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 41 Got response from kite 2021-02-26 18:19:12,326 [DEBUG] [spyder.plugins.completion.fallback.actor] -> Perform request textDocument/documentSymbol with id 41 2021-02-26 18:19:12,326 [DEBUG] [spyder.plugins.completion.snippets.actor] -> Perform request textDocument/documentSymbol with id 41 2021-02-26 18:19:12,526 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 40 timed out 2021-02-26 18:19:12,526 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 41 timed out 2021-02-26 18:19:12,569 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 40 Got response from lsp 2021-02-26 18:19:12,569 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 40 removed 2021-02-26 18:19:12,569 [DEBUG] [spyder.plugins.completion.plugin] -> Gather responses for textDocument/foldingRange ```

mrclary commented 3 years ago

Well, this is bizarre. I performed the tests on 4.2.1 and artifact from 14719 and got very poor results.

I'm wondering if it has something to do with the choice of keystroke cadence and intermission time. Could it be that sensitive to exactly which values I choose?

Why would I experience no issues typing in 4.2.1 or 14719, but with 4.2.2? Could my natural typing cadence be in poor resonance with 4.2.2 and not the others?

Would we be playing wack-a-mole by modifying timeouts, i.e. what works for some people causes issues for others? This seems to be an unstable prospect.

4.2.1

Very large latency at the intermission... sluggish_editor-4 2 1

PR #14719

This actually hangs indefinitely...I had to force quit Spyder. sluggish_editor_14719

mrclary commented 3 years ago

I forgot to mention that performing this test on 4.2.2 can cause Spyder to crash. server_python_47508.log transport_python_47508.log spyder.log

mrclary commented 3 years ago

With all the following preferences enabled simultaneously, and indent guides and code folding disabled, there are no issues whatsoever.

class and function selector
enable code completions
- show completion details (1000ms)
- show completions on the fly (1000ms)
enable basic linting
underline errors and warnings
introspection (all basic features)
enable code style linting
enable fallback completions

With either indent guides or code folding individually enabled, the issue manifests.

indent guides -> bad
code folding -> bad
indent guides + code folding -> bad + high probability of crashing.

mrclary commented 3 years ago

Testing with bootstrap is even worse than with the macOS application. Even with indent guides and code folding disabled, it is extremely sluggish and consistently crashes Spyder. This could be due to Anaconda Python builds vs. Homebrew Python builds; not sure.

pkabele commented 3 years ago

I have noticed the typing latency as described above in Spyder 4.2.3, Windows 10, Python 3.9.2. I was able to link the symptom to Kite running (no problem) or not running (typing is lagging). This happens regardless of the state of the switches in Completion and linting - Advanced - Providers - Enable Kite, Enable fallback completions being on or off.

leet0rz commented 3 years ago

This is also an issue in a blank file with nothing in it as long as Spyder stays open a few hours.

mrclary commented 3 years ago

Issue is still present in 5.1.0dev0 6648e8632 (latest 5.x, PR #15149). Bootstrap is very bad. Working with the artifact, it seems to still manifest only in large files

I've gotten a report from a colleague that Spyder 5.0.0 continues to get worse the longer it is open. He has downgraded to 4.2.5.

ccordoba12 commented 3 years ago

Bootstrap is very bad.

If you're in Big Sur and using the defaults Anaconda channel, we can say for sure that that is caused by Qt 5.9 and there's nothing we can do about it (that version is too old for Big Sur).

Working with the artifact, it seems to still manifest only in large files

We're aware of that problem and we'll fix it in the next months. It requires an important refactoring in the editor.

I've gotten a report from a colleague that Spyder 5.0.0 continues to get worse the longer it is open. He has downgraded to 4.2.5.

Worse in what sense?

mrclary commented 3 years ago

Bootstrap is very bad.

If you're in Big Sur and using the defaults Anaconda channel, we can say for sure that that is caused by Qt 5.9 and there's nothing we can do about it (that version is too old for Big Sur).

I'm reporting on Catalina, not Big Sur. But I can investigate more on Qt 5.9.

Working with the artifact, it seems to still manifest only in large files

We're aware of that problem and we'll fix it in the next months. It requires an important refactoring in the editor.

I've gotten a report from a colleague that Spyder 5.0.0 continues to get worse the longer it is open. He has downgraded to 4.2.5.

Worse in what sense?

This is anecdotal and I'm not prepared to quantify that. If we are refactoring the Editor soon, then I'd just as soon wait until then and revisit/retest this issue.

ccordoba12 commented 3 years ago

I'm reporting on Catalina, not Big Sur. But I can investigate more on Qt 5.9.

Ok, please check with 5.12 from Conda-forge. If the installer is working fine for you, I'd say that's the main issue.

lamorton commented 3 years ago

I just upgraded to Big Sur and hit this issue. I also updated conda/spyder, and it persists. I'm at:

Spyder 4.2.0 
Python 3.7.7 64-bit
Qt 5.9.6
PyQt5 5.9.2
Darwin 20.5.0

I am editing a short file (~20 lines). Kite is not installed. I unchecked code folding & indent guides: the issue persists.

mrclary commented 3 years ago

@lamorton, I recommend using the macOS app bundle instead of the conda version of Spyder. I find it is more responsive. Let me know if that helps at all. But the typing latency, in general, is still an issue.

mmacferrin commented 3 years ago

I, too, am having severe latency issues, here in Anaconda Spyder 5.1.1. I've updated whenever a new release comes out, and it keeps happening. The longer I keep the window open and keep editing, the worse (slower) it gets. I find myself closing Spyder and restarting it every 20-30 minutes just so I can keep working, which is a pain.

Ubuntu 20.04
Qt 5.12.9
conda 4.10.3 (installed from conda-forge)
Python 3.9.6, x64
pyqt 5.12.13
spyder 5.1.1

I'm on a 20-core Xeon machine with plenty of RAM, it's not even close to maxing out the machine, but it just starts spinning indefinitely on the one processor for reasons unknown. Doesn't seem like a text editor should be doing this (even with Kite, code-completion, etc). I am trying to see if downgrading spyder might help for a while, but if I can't solve it I'm likely just going to need to migrate to another programming environment and leave spyder behind. This is not a sustainable way to work, pressing keys and waiting around 5+ seconds for them to appear on the screen before going on. The Commodore 64's of my childhood were more responsive than this.

mrclary commented 3 years ago

Current Status

Spyder version: 5.1.5
Operating System: Darwin 20.6.0

Test script

```python #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Test Spyder typing latency """ import time, keyboard time.sleep(3) t0 = time.time_ns() keyboard.write('The quick brown fox ', delay=0.05) t1 = time.time_ns() time.sleep(1.5) t2 = time.time_ns() keyboard.write('jumped over the lazy dog.', delay=0.05) t3 = time.time_ns() dt1 = (t1 - t0) / 1e6 dt2 = (t3 - t2) / 1e6 print(f'Fast = {dt1:0.0f} ms; Slow = {dt2:0.0f} ms; Slow - Fast = {dt2-dt1:0.0f} ms') ```

Using spyder/app/mainwindow.py (~2000 lines). Both applications open for less than 20min and only one file open. Code folding and indent guides enabled.

Bootstrap

bootstrap

Bootstrap conda environment

``` Python 3.9.6 (default, Aug 18 2021, 12:38:10) [Clang 10.0.0 ] # packages in environment at /Users/rclary/opt/miniconda3/envs/spy-dev: # # Name Version Build Channel alabaster 0.7.12 pyhd3eb1b0_0 altgraph 0.17 pyhd3eb1b0_0 appdirs 1.4.4 pyhd3eb1b0_0 applaunchservices 0.2.1 pyhd3eb1b0_0 appnope 0.1.2 py39hecd8cb5_1001 argh 0.26.2 py39hecd8cb5_0 arrow 0.13.1 py39hecd8cb5_0 astroid 2.6.6 py39hecd8cb5_0 async_generator 1.10 pyhd3eb1b0_0 atomicwrites 1.4.0 py_0 attrs 21.2.0 pyhd3eb1b0_0 autopep8 1.5.7 pyhd3eb1b0_0 babel 2.9.1 pyhd3eb1b0_0 backcall 0.2.0 pyhd3eb1b0_0 bcrypt 3.2.0 py39h9ed2024_0 binaryornot 0.4.4 pyhd3eb1b0_1 black 19.10b0 py_0 blas 1.0 mkl bleach 4.0.0 pyhd3eb1b0_0 bottleneck 1.3.2 py39he3068b8_1 brotli 1.0.9 hb1e8313_2 brotlipy 0.7.0 py39h9ed2024_1003 ca-certificates 2021.7.5 hecd8cb5_1 certifi 2021.5.30 py39hecd8cb5_0 cffi 1.14.6 py39h2125817_0 chardet 4.0.0 py39hecd8cb5_1003 charset-normalizer 2.0.4 pyhd3eb1b0_0 click 8.0.1 pyhd3eb1b0_0 cloudpickle 1.6.0 pyhd3eb1b0_0 cookiecutter 1.7.2 pyhd3eb1b0_0 coverage 5.5 py39h9ed2024_2 cryptography 3.4.7 py39h2fd3fbb_0 cycler 0.10.0 py39hecd8cb5_0 cython 0.29.24 py39h23ab428_0 debugpy 1.4.1 py39h23ab428_0 decorator 5.0.9 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 diff-match-patch 20200713 pyhd3eb1b0_0 docutils 0.17.1 py39hecd8cb5_1 entrypoints 0.3 py39hecd8cb5_0 flake8 3.9.2 pyhd3eb1b0_0 flaky 3.7.0 py_0 fonttools 4.25.0 pyhd3eb1b0_0 freetype 2.10.4 ha233b18_0 gmp 6.2.1 h23ab428_2 gmpy2 2.0.8 py39h4b98818_3 icu 58.2 h0a44026_3 idna 3.2 pyhd3eb1b0_0 imagesize 1.2.0 pyhd3eb1b0_0 importlib-metadata 4.6.4 py39hecd8cb5_0 importlib_metadata 4.6.4 hd3eb1b0_0 inflection 0.5.1 py39hecd8cb5_0 intel-openmp 2021.3.0 hecd8cb5_3375 intervaltree 3.1.0 pyhd3eb1b0_0 ipykernel 6.2.0 py39hecd8cb5_1 ipython 7.26.0 py39h01d92e1_0 ipython_genutils 0.2.0 pyhd3eb1b0_1 isort 5.9.3 pyhd3eb1b0_0 jedi 0.18.0 py39hecd8cb5_1 jinja2 2.11.3 pyhd3eb1b0_0 jinja2-time 0.2.0 pyhd3eb1b0_2 jpeg 9b he5867d9_2 jsonschema 3.2.0 pyhd3eb1b0_2 jupyter-client 6.1.12 pypi_0 pypi jupyter_core 4.7.1 py39hecd8cb5_0 jupyterlab_pygments 0.1.2 py_0 keyring 23.0.1 py39hecd8cb5_0 kiwisolver 1.3.1 py39h23ab428_0 lazy-object-proxy 1.6.0 py39h9ed2024_0 lcms2 2.12 hf1fd2bf_0 libcxx 12.0.0 h2f01273_0 libffi 3.3 hb1e8313_2 libgfortran 3.0.1 h93005f0_2 libpng 1.6.37 ha441bb4_0 libsodium 1.0.18 h1de35cc_0 libspatialindex 1.9.3 h23ab428_0 libtiff 4.2.0 h87d7836_0 libwebp-base 1.2.0 h9ed2024_0 lz4-c 1.9.3 h23ab428_1 macholib 1.14 pyhd3eb1b0_1 markupsafe 1.1.1 py39h9ed2024_0 matplotlib 3.4.2 py39hecd8cb5_0 matplotlib-base 3.4.2 py39h8b3ea08_0 matplotlib-inline 0.1.2 pyhd3eb1b0_2 mccabe 0.6.1 py39hecd8cb5_1 mistune 0.8.4 py39h9ed2024_1000 mkl 2021.3.0 hecd8cb5_517 mkl-service 2.4.0 py39h9ed2024_0 mkl_fft 1.3.0 py39h4a7008c_2 mkl_random 1.2.2 py39hb2f4e1b_0 modulegraph 0.19.2 pyhd8ed1ab_0 conda-forge more-itertools 8.8.0 pyhd3eb1b0_0 mpc 1.1.0 h6ef4df4_1 mpfr 4.0.2 h9066e36_1 mpmath 1.2.1 py39hecd8cb5_0 munkres 1.1.4 py_0 mypy_extensions 0.4.3 py39hecd8cb5_0 nbclient 0.5.3 pyhd3eb1b0_0 nbconvert 6.1.0 py39hecd8cb5_0 nbformat 5.1.3 pyhd3eb1b0_0 ncurses 6.2 h0a44026_1 nest-asyncio 1.5.1 pyhd3eb1b0_0 numexpr 2.7.3 py39h5873af2_1 numpy 1.20.3 py39h4b4dc7a_0 numpy-base 1.20.3 py39he0bd621_0 numpydoc 1.1.0 pyhd3eb1b0_1 olefile 0.46 pyhd3eb1b0_0 openjpeg 2.4.0 h66ea3da_0 openssl 1.1.1l h9ed2024_0 packaging 21.0 pyhd3eb1b0_0 pandas 1.3.2 py39h5008ddb_0 pandocfilters 1.4.3 py39hecd8cb5_1 paramiko 2.7.2 py_0 parso 0.8.2 pyhd3eb1b0_0 pathspec 0.7.0 py_0 pexpect 4.8.0 pyhd3eb1b0_3 pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 8.3.1 py39ha4cf6ea_0 pip 21.2.4 py37hecd8cb5_0 pluggy 0.13.1 py39hecd8cb5_0 poyo 0.5.0 pyhd3eb1b0_0 prompt-toolkit 3.0.17 pyhca03da5_0 psutil 5.8.0 py39h9ed2024_1 ptyprocess 0.7.0 pyhd3eb1b0_2 py 1.10.0 pyhd3eb1b0_0 py2app 0.22 py39hde42818_1 conda-forge pycodestyle 2.7.0 pyhd3eb1b0_0 pycparser 2.20 py_2 pydocstyle 6.1.1 pyhd3eb1b0_0 pyflakes 2.3.1 pyhd3eb1b0_0 pygments 2.10.0 pyhd3eb1b0_0 pylint 2.9.6 py39hecd8cb5_1 pyls-spyder 0.4.0 pyhd8ed1ab_0 conda-forge pynacl 1.4.0 py39h9ed2024_1 pyopenssl 20.0.1 pyhd3eb1b0_1 pyparsing 2.4.7 pyhd3eb1b0_0 pyqt 5.9.2 py39h23ab428_6 pyrsistent 0.18.0 py39h9ed2024_0 pysocks 1.7.1 py39hecd8cb5_0 pytest 5.4.3 py39hecd8cb5_0 pytest-cov 2.12.1 pyhd3eb1b0_0 pytest-lazy-fixture 0.6.3 py_0 spyder-ide pytest-mock 3.6.1 pyhd3eb1b0_0 pytest-order 0.10.0 pyhd8ed1ab_0 spyder-ide pytest-qt 3.3.0 py_0 spyder-ide python 3.9.6 h88f2d9e_1 python-dateutil 2.8.2 pyhd3eb1b0_0 python-lsp-black 1.0.0 pyhd8ed1ab_0 conda-forge python-lsp-jsonrpc 1.0.0 pyhd8ed1ab_0 conda-forge python-slugify 5.0.2 pyhd3eb1b0_0 python.app 3 py39h9ed2024_0 python_abi 3.9 2_cp39 conda-forge pytz 2021.1 pyhd3eb1b0_0 pyxdg 0.27 pyhd3eb1b0_0 pyyaml 5.4.1 py39h9ed2024_1 pyzmq 22.2.1 py39h23ab428_1 qdarkstyle 3.0.2 dev_0 qstylizer 0.1.10 pyhd3eb1b0_0 qt 5.9.7 h468cd18_1 qtawesome 1.0.2 pyhd3eb1b0_0 qtconsole 5.1.0 pyhd3eb1b0_0 qtpy 1.10.0 pyhd3eb1b0_0 readline 8.1 h9ed2024_0 regex 2021.8.3 py39h9ed2024_0 requests 2.26.0 pyhd3eb1b0_0 rope 0.19.0 pyhd3eb1b0_0 rtree 0.9.7 py39hecd8cb5_1 scipy 1.7.1 py39h88652d9_2 setuptools 52.0.0 py39hecd8cb5_0 sip 4.19.13 py39h23ab428_0 six 1.16.0 pyhd3eb1b0_0 snowballstemmer 2.1.0 pyhd3eb1b0_0 sortedcontainers 2.4.0 pyhd3eb1b0_0 sphinx 4.0.2 pyhd3eb1b0_0 sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0 sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0 sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0 sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0 sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0 sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0 spyder-kernels 2.2.0.dev0 dev_0 sqlite 3.36.0 hce871da_0 sympy 1.8 py39hecd8cb5_0 testpath 0.5.0 pyhd3eb1b0_0 text-unidecode 1.3 pyhd3eb1b0_0 textdistance 4.2.1 pyhd3eb1b0_0 three-merge 0.1.1 pyhd3eb1b0_0 tinycss 0.4 pyhd3eb1b0_1002 tk 8.6.10 hb0a8c7a_0 toml 0.10.2 pyhd3eb1b0_0 tornado 6.1 py39h9ed2024_0 traitlets 5.0.5 pyhd3eb1b0_0 typed-ast 1.4.3 py39h9ed2024_1 typing_extensions 3.10.0.0 pyhca03da5_0 tzdata 2021a h5d7bf9c_0 ujson 4.0.2 py39h23ab428_0 unidecode 1.2.0 pyhd3eb1b0_0 urllib3 1.26.6 pyhd3eb1b0_1 watchdog 2.1.3 py39hd5f6b7e_0 wcwidth 0.2.5 pyhd3eb1b0_0 webencodings 0.5.1 py39hecd8cb5_1 wheel 0.37.0 pyhd3eb1b0_1 whichcraft 0.6.1 pyhd3eb1b0_0 wrapt 1.12.1 py39h9ed2024_1 wurlitzer 2.1.1 py39hecd8cb5_0 xz 5.2.5 h1de35cc_0 yaml 0.2.5 haf1e3a3_0 yapf 0.31.0 pyhd3eb1b0_0 zeromq 4.3.4 h23ab428_0 zipp 3.5.0 pyhd3eb1b0_0 zlib 1.2.11 h1de35cc_3 zstd 1.4.9 h322a384_0 ```

macOS App

macOS-app

macOS app build environment

``` Python 3.9.5 (default, May 25 2021, 07:41:55) [Clang 11.0.0 (clang-1100.0.33.17)] Package Version ------------------------------------------------- --------- alabaster 0.7.12 altgraph 0.17.2 applaunchservices 0.2.1 appnope 0.1.2 arrow 1.1.1 astroid 2.6.6 atomicwrites 1.4.0 attrs 21.2.0 autopep8 1.5.7 Babel 2.9.1 backcall 0.2.0 bcrypt 3.2.0 binaryornot 0.4.4 black 21.9b0 bleach 4.1.0 certifi 2021.5.30 cffi 1.14.6 chardet 4.0.0 charset-normalizer 2.0.5 click 8.0.1 cloudpickle 2.0.0 cookiecutter 1.7.3 cryptography 3.4.8 cycler 0.10.0 Cython 0.29.24 debugpy 1.4.3 decorator 5.1.0 defusedxml 0.7.1 diff-match-patch 20200713 dmgbuild 1.5.2 docutils 0.17.1 ds-store 1.3.0 entrypoints 0.3 flake8 3.9.2 idna 3.2 imagesize 1.2.0 importlib-metadata 4.8.1 inflection 0.5.1 intervaltree 3.1.0 ipykernel 6.4.1 ipython 7.27.0 ipython-genutils 0.2.0 isort 5.9.3 jedi 0.18.0 Jinja2 3.0.1 jinja2-time 0.2.0 jsonschema 3.2.0 jupyter-client 6.1.12 jupyter-core 4.7.1 jupyterlab-pygments 0.1.2 keyring 23.2.1 kiwisolver 1.3.2 lazy-object-proxy 1.6.0 mac-alias 2.2.0 macholib 1.15.2 MarkupSafe 2.0.1 matplotlib 3.4.3 matplotlib-inline 0.1.3 mccabe 0.6.1 mistune 0.8.4 modulegraph 0.19.2 mpmath 1.2.1 mypy-extensions 0.4.3 nbclient 0.5.4 nbconvert 6.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 numpy 1.21.2 numpydoc 1.1.0 packaging 21.0 pandas 1.3.3 pandocfilters 1.5.0 paramiko 2.7.2 parso 0.8.2 pathspec 0.9.0 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.3.2 pip 21.2.4 platformdirs 2.3.0 pluggy 1.0.0 poyo 0.5.0 prompt-toolkit 3.0.20 psutil 5.8.0 ptyprocess 0.7.0 py2app 0.22 pycodestyle 2.7.0 pycparser 2.20 pydocstyle 6.1.1 pyflakes 2.3.1 Pygments 2.10.0 pylint 2.9.6 pyls-spyder 0.4.0 PyNaCl 1.4.0 pyobjc 7.3 pyobjc-core 7.3 pyobjc-framework-Accounts 7.3 pyobjc-framework-AddressBook 7.3 pyobjc-framework-AdSupport 7.3 pyobjc-framework-AppleScriptKit 7.3 pyobjc-framework-AppleScriptObjC 7.3 pyobjc-framework-ApplicationServices 7.3 pyobjc-framework-AuthenticationServices 7.3 pyobjc-framework-AutomaticAssessmentConfiguration 7.3 pyobjc-framework-Automator 7.3 pyobjc-framework-AVFoundation 7.3 pyobjc-framework-AVKit 7.3 pyobjc-framework-BusinessChat 7.3 pyobjc-framework-CalendarStore 7.3 pyobjc-framework-CFNetwork 7.3 pyobjc-framework-CloudKit 7.3 pyobjc-framework-Cocoa 7.3 pyobjc-framework-Collaboration 7.3 pyobjc-framework-ColorSync 7.3 pyobjc-framework-Contacts 7.3 pyobjc-framework-ContactsUI 7.3 pyobjc-framework-CoreAudio 7.3 pyobjc-framework-CoreAudioKit 7.3 pyobjc-framework-CoreBluetooth 7.3 pyobjc-framework-CoreData 7.3 pyobjc-framework-CoreHaptics 7.3 pyobjc-framework-CoreLocation 7.3 pyobjc-framework-CoreMedia 7.3 pyobjc-framework-CoreMediaIO 7.3 pyobjc-framework-CoreMIDI 7.3 pyobjc-framework-CoreML 7.3 pyobjc-framework-CoreMotion 7.3 pyobjc-framework-CoreServices 7.3 pyobjc-framework-CoreSpotlight 7.3 pyobjc-framework-CoreText 7.3 pyobjc-framework-CoreWLAN 7.3 pyobjc-framework-CryptoTokenKit 7.3 pyobjc-framework-DeviceCheck 7.3 pyobjc-framework-DictionaryServices 7.3 pyobjc-framework-DiscRecording 7.3 pyobjc-framework-DiscRecordingUI 7.3 pyobjc-framework-DiskArbitration 7.3 pyobjc-framework-DVDPlayback 7.3 pyobjc-framework-EventKit 7.3 pyobjc-framework-ExceptionHandling 7.3 pyobjc-framework-ExecutionPolicy 7.3 pyobjc-framework-ExternalAccessory 7.3 pyobjc-framework-FileProvider 7.3 pyobjc-framework-FileProviderUI 7.3 pyobjc-framework-FinderSync 7.3 pyobjc-framework-FSEvents 7.3 pyobjc-framework-GameCenter 7.3 pyobjc-framework-GameController 7.3 pyobjc-framework-GameKit 7.3 pyobjc-framework-GameplayKit 7.3 pyobjc-framework-ImageCaptureCore 7.3 pyobjc-framework-IMServicePlugIn 7.3 pyobjc-framework-InputMethodKit 7.3 pyobjc-framework-InstallerPlugins 7.3 pyobjc-framework-InstantMessage 7.3 pyobjc-framework-Intents 7.3 pyobjc-framework-IOSurface 7.3 pyobjc-framework-iTunesLibrary 7.3 pyobjc-framework-LatentSemanticMapping 7.3 pyobjc-framework-LaunchServices 7.3 pyobjc-framework-libdispatch 7.3 pyobjc-framework-LinkPresentation 7.3 pyobjc-framework-LocalAuthentication 7.3 pyobjc-framework-MapKit 7.3 pyobjc-framework-MediaAccessibility 7.3 pyobjc-framework-MediaLibrary 7.3 pyobjc-framework-MediaPlayer 7.3 pyobjc-framework-MediaToolbox 7.3 pyobjc-framework-Metal 7.3 pyobjc-framework-MetalKit 7.3 pyobjc-framework-MetalPerformanceShaders 7.3 pyobjc-framework-ModelIO 7.3 pyobjc-framework-MultipeerConnectivity 7.3 pyobjc-framework-NaturalLanguage 7.3 pyobjc-framework-NetFS 7.3 pyobjc-framework-Network 7.3 pyobjc-framework-NetworkExtension 7.3 pyobjc-framework-NotificationCenter 7.3 pyobjc-framework-OpenDirectory 7.3 pyobjc-framework-OSAKit 7.3 pyobjc-framework-OSLog 7.3 pyobjc-framework-PencilKit 7.3 pyobjc-framework-Photos 7.3 pyobjc-framework-PhotosUI 7.3 pyobjc-framework-PreferencePanes 7.3 pyobjc-framework-PushKit 7.3 pyobjc-framework-Quartz 7.3 pyobjc-framework-QuickLookThumbnailing 7.3 pyobjc-framework-SafariServices 7.3 pyobjc-framework-SceneKit 7.3 pyobjc-framework-ScreenSaver 7.3 pyobjc-framework-ScriptingBridge 7.3 pyobjc-framework-SearchKit 7.3 pyobjc-framework-Security 7.3 pyobjc-framework-SecurityFoundation 7.3 pyobjc-framework-SecurityInterface 7.3 pyobjc-framework-ServiceManagement 7.3 pyobjc-framework-Social 7.3 pyobjc-framework-SoundAnalysis 7.3 pyobjc-framework-Speech 7.3 pyobjc-framework-SpriteKit 7.3 pyobjc-framework-StoreKit 7.3 pyobjc-framework-SyncServices 7.3 pyobjc-framework-SystemConfiguration 7.3 pyobjc-framework-SystemExtensions 7.3 pyobjc-framework-UserNotifications 7.3 pyobjc-framework-VideoSubscriberAccount 7.3 pyobjc-framework-VideoToolbox 7.3 pyobjc-framework-Vision 7.3 pyobjc-framework-WebKit 7.3 pyparsing 2.4.7 PyQt5 5.12.3 PyQt5-sip 12.9.0 PyQtWebEngine 5.12.1 pyrsistent 0.18.0 python-dateutil 2.8.2 python-lsp-black 1.0.0 python-lsp-jsonrpc 1.0.0 python-lsp-server 1.2.2 python-slugify 5.0.2 pytz 2021.1 pyxdg 0.27 pyzmq 22.2.1 QDarkStyle 3.0.2 qstylizer 0.2.1 QtAwesome 1.0.3 qtconsole 5.1.1 QtPy 1.11.1 regex 2021.8.28 requests 2.26.0 rope 0.19.0 Rtree 0.9.7 scipy 1.7.1 setuptools 58.0.4 six 1.16.0 snowballstemmer 2.1.0 sortedcontainers 2.4.0 Sphinx 4.2.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 spyder-kernels 2.1.1 sympy 1.8 testpath 0.5.0 text-unidecode 1.3 textdistance 4.2.1 three-merge 0.1.1 tinycss2 1.1.0 toml 0.10.2 tomli 1.2.1 tornado 6.1 traitlets 5.1.0 typing-extensions 3.10.0.2 ujson 4.1.0 urllib3 1.26.6 watchdog 2.1.5 wcwidth 0.2.5 webencodings 0.5.1 wrapt 1.12.1 wurlitzer 3.0.2 yapf 0.31.0 zipp 3.5.0 ```

ccordoba12 commented 3 years ago

@mmacferrin, most latency issues are fixed in our latest version (5.1.5). Please update.

@mrclary, the remaining issues are due to code folding and indent guides, which require a big refactoring in the editor to fix them. Please disable them and try again.

mrclary commented 3 years ago

@ccordoba12, I can confirm that disabling both code folding and indent guides eliminates the issue, as far as I can tell.

With them enabled, playing around with SYNC_SYMBOLS_AND_FOLDING_TIMEOUTS and testing on spyder/plugins/editor/widgets/codeeditor.py (~5500 lines), I see that if I set the timeout to less than 1500ms (the pause delay in my testing script) then the issue manifests. If I set it to larger than 1500ms, the issue does not manifest. This confirms your assertion that the issue is with the performance of the folding algorithm (indent guides depend on folding). Since a user's "delay" could be just about anything, the timeout cannot be used as a solution, but the performance of the folding must be improved considerably.

Indeed, spyder/plugins/editor/panels/utils.py::merge_folding takes 13-16s to complete! Ouch! It is the while loop in that function that is taking so long. For the testing algorithm I used, the loop iterates 1580 times. Within the loop, it is textdistance.jaccard.normalized_similarity that accounts for the entire accumulated loop time.

mrclary commented 3 years ago

I think I've got a very good and simple fix. I'll submit a PR shortly

mrclary commented 3 years ago

So, I think this can be solved by changing the algorithm used by textdistance. Doing some experimentation, I found a few options:

Algorithm	Test time (s)	Loop size	Dist > 0.8	Note
jaccard	~15	1580	14	this is what we currently use
tanimoto	~0.11	1580	0	supposedly the same as jaccard but with different parameters
overlap	~0.08	1580	1333
hamming	~0.01	1580	14	requires jellyfish optimization package available on conda

I recommend using the hanning because it is by far the fastest, but also because it has the same number of iterations as our current algorithm that have a distance measurements above 0.8. This means that it is probably similar in its comparison of the texts. However, textdistance documentation suggests that jaccard is "token" based and hanning is edit_based, whatever that means. This may or may not make any difference to us.

The bottom line is that with these changes we can probably have one timeout, independent of the number of lines, and the typing latency should be solved without refactoring. :-)

spyder-ide / spyder