Python LSP server completely stops working after a cancel notification is received

[hugorichard]

I installed pylsp and all required modules. When using python-lsp-server I see the following warning "2021-11-14 15:12:47,855 CET - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 10" and then everything stops working : the auto-completions, linting, signature help.

I am a spacemacs user (=emacs). A lot of other users (including some people using different IDE) have experienced this issue. See: https://github.com/palantir/python-jsonrpc-server/issues/17 https://github.com/python-lsp/python-lsp-jsonrpc/issues/5 https://github.com/emacs-lsp/lsp-mode/issues/1731 https://github.com/palantir/python-jsonrpc-server/issues/17

Any ideas on what could be the problem ?

Best, Hugo

[m0wer]

Same here from using nvim 0.6.1 with the following config:

nvim_lsp.pylsp.setup(require('coq').lsp_ensure_capabilities({
    settings = {
        pylsp = {
            configurationSources = {'flake8', 'pycodestyle'},
            plugins = {
                jedi_completion = {
                    fuzzy = true,
                },
                flake8 = {
                    enabled = true,
                    config = ".flake8",
                },
                pycodestyle = {
                    enabled = true,
                },
                mccabe = {
                    enabled = true,
                },
                preload = {
                    enabled = false,
                },
                pydocstyle = {
                    enabled = true,
                },
                pylint = {
                    enabled = false,
                },
                pyflakes = {
                    enabled = false,
                },
                rope_completion = {
                    enabled = true,
                },
                yapf = {
                    enabled = false,
                },
            },
        },
    },
    flags = {
      debounce_text_changes = 150,
    },
  }))

The process pylsp uses 100% of CPU and logs (~/.cache/nvim/lsp.log) show something like this:

[ERROR][2022-06-14 10:55:40] .../vim/lsp/rpc.lua:420    "rpc"   "pylsp" "stderr"    "2022-06-14 10:55:40,342 +03 - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 129\n"
[ERROR][2022-06-14 10:55:46] .../vim/lsp/rpc.lua:420    "rpc"   "pylsp" "stderr"    "2022-06-14 10:55:46,355 +03 - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 130\n"
[ERROR][2022-06-14 10:55:50] .../vim/lsp/rpc.lua:420    "rpc"   "pylsp" "stderr"    "2022-06-14 10:55:50,056 +03 - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 131\n"
[ERROR][2022-06-14 10:55:53] .../vim/lsp/rpc.lua:420    "rpc"   "pylsp" "stderr"    "2022-06-14 10:55:53,421 +03 - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 132\n"

[ccordoba12]

@m0wer, please describe step by step how to set up a Neovim instance with LSP and this server working. That way I could reproduce this error on my side and then try to fix it.

[Noborita9]

Hi, I started having this problem yesterday, I updated my plugins with PackerSync and the python-language-server started being slow, in just around 20s it stops sending suggestions, it just gives syntax errors and its super slow. If any more information is needed i would like to contribute to solve this

[satyajeet-sundial]

@ccordoba12 this is quite easily reproducible in emacs with lsp-mode with quite a minimal setup

[satyajeet-sundial]

This is what the sample log lines in my log looks like:

2023-05-17 11:17:33,077 IST - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 4073
2023-05-17 11:17:33,106 IST - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 4074
2023-05-17 11:17:34,681 IST - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 4072
2023-05-17 11:17:34,698 IST - WARNING - pylsp_jsonrpc.endpoint - Received cancel notification for unknown message id 4078

[carlos-gtz]

I'm having the same issue with neovim nvim-lspconfig. All those "WARNING - pylsp_jsonrpc.endpoint" messages are being spammed on lsp log

My configuration is:

{
   settings = {
      pylsp = {
         plugins = {
            configurationSources = { "flake8" },

            autopep8 = { enabled = false },
            flake8 = { enabled = true },
            mccabe = { enabled = false },
            preload = { enabled = false },
            pycodestyle = { enabled = false },
            pydocstyle = { enabled = false },
            pyflakes = { enabled = false },
            pylint = { enabled = false },
            yapf = { enabled = false },

            -- https://github.com/python-rope/pylsp-rope
            rope_auto_import = { enabled = false, memory = false },
            rope_completion = { enabled = false, eager = false },

            -- 3rd party plugins
            -- https://github.com/Richardk2n/pylsp-mypy
            pylsp_mypy = {
               live_mode = false,
               report_progress = true,
               dmypy = true,
            },

            isort = { enabled = true },
            black = { enabled = true, cache_config = true },
            ruff = { enabled = true },

            -- Jedi configuration
            jedi_hover = { enabled = true },
            jedi_signature_help = { enabled = true },
            jedi_references = { enabled = true },
            jedi_symbols = { enabled = true },
            jedi_definition = {
               enabled = true,
               follow_imports = true,
               follow_builtin_imports = true,
               follow_builtin_definitions = true,
            },
            jedi_completion = {
               enabled = true,
               include_params = true,
               include_class_objects = false,
               include_function_objects = false,
               fuzzy = false,
               eager = false,
               resolve_at_most = 25,
               cache_for = {
                  "pandas",
                  "numpy",
                  "tensorflow",
                  "matplotlib",
                  "pydantic",
                  "fastapi",
                  "flask",
                  "sqlalchemy",
                  "dagster",
               },
            },
         },
      },
   },
}

[malashinroman]

same issue with neovim 0.8.3 or 0.9.1

[fecet]

same issue with neovim nightly

[ccordoba12]

We'll see what we can do about this problem in our next version.

[smacke]

@ccordoba12 I was able to get this warning message in logs, but I wasn't able to reproduce some of the issues others described around slowing performance or logs getting flooded. At a high level, though, it seems like it must be related to clients that don't throttle requests because they assume they can be cancelled. Whenever I saw Received cancel notification for unknown message id, it was always because the client attempted to cancel a request with a synchronous / blocking handler, and therefore not cancellable.

The most straightforward solution I see is to implement server-side throttling in python-lsp-jsonrpc. Here's a rough description of the design I'm thinking of. For each request type with a synchronous handler (e.g. codeLens, codeAction, etc), we keep track of the timestamp of the last time the handler was called. (One possibility could be to make throttling opt-in per request type, but I think we could probably just do it for all of them.) If another request of the same type comes in within epsilon of the last timestamp, then we schedule it to run epsilon time units in the future, where epsilon has some sensible default but can be configured. If there was already some request of that type scheduled to run, we cancel it and respond with THROTTLED or whatever standard thing json rpc gives us, and schedule a run of the new request that came in for the request type.

Then the question remains of what to do about the log messages, e.g. for client attempts to cancel a non-cancellable request, but also for logging warnings when requests are getting throttled. I think it probably makes sense to keep the log level at warning, because in my opinion, it indicates a client misbehavior that the client library maintainer has a responsibility to fix. However, we can also introduce another layer of throttling within the logger itself, so that we at least aren't flooding logs.

I think all of this throttling logic probably belongs in python-lsp-jsonrpc in the Endpoint class. For now, I'm thinking we can just throttle requests and not notifications, since most reasonable clients are hopefully already throttling or debouncing notifications, e.g. document/didChange.

How does this sound do you? If it sounds OK, I'd be happy to take a stab at contributing an implementation.

[ccordoba12]

The most straightforward solution I see is to implement server-side throttling in python-lsp-jsonrpc. Here's a rough description of the design I'm thinking of.

Thanks for your careful thinking about this @smacke! Your proposed plan sounds good to me.

Then the question remains of what to do about the log messages, e.g. for client attempts to cancel a non-cancellable request, but also for logging warnings when requests are getting throttled. I think it probably makes sense to keep the log level at warning, because in my opinion, it indicates a client misbehavior that the client library maintainer has a responsibility to fix.

Agreed.

However, we can also introduce another layer of throttling within the logger itself, so that we at least aren't flooding logs.

Agreed too.

I think all of this throttling logic probably belongs in python-lsp-jsonrpc in the Endpoint class.

Yep, I think so too.

How does this sound do you? If it sounds OK, I'd be happy to take a stab at contributing an implementation.

Sounds good to me. My only question is this: what if we make the server fully async? (there's python-lsp/python-lsp-jsonrpc#12 as a starting point). I mean, I don't know if this problem would be automatically solved if we do that instead.

[mokazemi]

I'm also getting this warning on Kate editor.

[Cherkah]

as many people i'm getting this warning (kde kate editor)