search

microsoft / pylance-release

Documentation and issues for Pylance
Creative Commons Attribution 4.0 International
1.71k stars 765 forks source link

How to set pylance memory max limitation? #5799

Open Mactarvish opened 6 months ago

Mactarvish commented 6 months ago

Pylance works fine when workspace is small but crashed when huge. Pylance has crashed. Would you like to try using a Node.js executable to run the language server?

Report issue is 

2024-04-25 11:25:35.477 [info] (Client) Pylance async client (2024.4.1) started with python extension (2024.4.1)
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Pylance language server 2024.4.1 (pyright version 1.1.356, commit 6652c4a8) starting
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Server root directory: file:///home/mck/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist
2024-04-25 11:25:36.064 [info] [Info  - 11:25:36 AM] (1667316) Starting service instance "lrcn"
2024-04-25 11:25:36.117 [info] [Info  - 11:25:36 AM] (1667316) Setting environmentName for service "lrcn": "D:\anaconda3\python.exe"
2024-04-25 11:25:46.295 [info] [Error - 11:25:46 AM] (1667316) Enumeration of workspace source files is taking longer than 10 seconds.
This may be because:
* You have opened your home directory or entire hard drive as a workspace
* Your workspace contains a very large number of directories and files
* Your workspace contains a symlink to a directory with many files
* Your workspace is remote, and file enumeration is slow
To reduce this time, open a workspace directory with fewer files or add a pyrightconfig.json configuration file with an "exclude" section to exclude subdirectories from your workspace. For more details, refer to https://github.com/microsoft/pyright/blob/main/docs/configuration.md.
2024-04-25 11:27:50.685 [info] 
<--- Last few GCs --->

[1667316:0x5c6aa10]   132258 ms: Mark-sweep 4055.7 (4139.7) -> 4042.8 (4142.5) MB, 966.2 / 0.0 ms  (average mu = 0.340, current mu = 0.057) allocation failure; scavenge might not succeed
[1667316:0x5c6aa10]   135181 ms: Mark-sweep 4058.5 (4142.5) -> 4045.5 (4144.7) MB, 2862.7 / 0.0 ms  (average mu = 0.158, current mu = 0.021) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

2024-04-25 11:27:50.686 [info]  1: 0xb85bc0 node::Abort() [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.686 [info]  2: 0xa94834  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.687 [info]  3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.687 [info]  4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.688 [info]  5: 0xf447c5  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.688 [info]  6: 0xf56cad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.689 [info]  7: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.689 [info]  8: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.690 [info]  9: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.691 [info] 10: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:50.691 [info] 11: 0x1705b39  [/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node]

2024-04-25 11:27:51.210 [info] [Error - 11:27:51 AM] Server process exited with signal SIGABRT.
2024-04-25 11:27:58.346 [info] [Error - 11:27:58 AM] Notify file events failed.
2024-04-25 11:27:58.346 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
    at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
    at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:27:59.034 [info] [Error - 11:27:59 AM] Notify file events failed.
2024-04-25 11:27:59.034 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
    at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
    at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:25.819 [info] [Error - 11:28:25 AM] Request textDocument/codeAction failed.
2024-04-25 11:28:25.819 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2310379)
    at _.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2351280)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async N.provideCodeActions (/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/out/vs/workbench/api/node/extensionHostProcess.js:154:95620)
2024-04-25 11:28:25.961 [info] [Error - 11:28:25 AM] Notify file events failed.
2024-04-25 11:28:25.961 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
    at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
    at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:26.557 [info] [Error - 11:28:26 AM] Notify file events failed.
2024-04-25 11:28:26.557 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308918)
    at _.sendNotification (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2352539)
    at async t.Delayer.<anonymous> (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2364453)
2024-04-25 11:28:29.889 [info] [Error - 11:28:29 AM] Request textDocument/definition failed.
2024-04-25 11:28:29.890 [info] Error: Connection is disposed.
    at pe (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2308075)
    at Object.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2310379)
    at _.sendRequest (/home/mck/.vscode-server/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:2351280)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async h.provideDefinition (/home/mck/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/out/vs/workbench/api/node/extensionHostProcess.js:154:92639)

As shown above : FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

But I have 256G memory on my server which is enough for pylance. How to set the max limitation of memory occupy?

Mactarvish commented 6 months ago

Otherwise, if there anyway to let pylance NOT analysis special directory or soft link? I've found that when I create a soft link to my HUGE dataset that causes pylance crash.

rchiodo commented 6 months ago

There is an exclude option:

https://github.com/microsoft/pyright/blob/main/docs/configuration.md

rchiodo commented 6 months ago

Additionally see the directions here: https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#pylance-is-crashing

rchiodo commented 6 months ago

However, we'd rather just fix your memory issue. Can you share your source so we can try and fix the root cause of the problem?

rchiodo commented 6 months ago

I tried an experiment with a recursive symlinked folder.

Folder looked like this without linking:

- non_linked
  - subfolder
    - __init__.py
    - funcs.py 
  - __init__.py
- importer.py

Where importer.py looks like this:

from exporter import *

And funcs.py looks like this:

from recursive import *
def foo_func():
   return 4

And the __init__.py files just export all the symbols inside of them.

Then I symlinked non_linked to exporter and inside of subfolder I added a recursive link to non_linked.

Running this in Pylance I get indexing results like so:

2024-04-25 23:28:56.272 [info] (83551) [IDX(3)]   index execution environment file:///home/rich/tests/test_symlinks [found 1218 in 17 files] (14ms)

With output like so:


[BG(1)]                                               binding: file:///home/rich/tests/test_symlinks/non_linked/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/subfolder/recursive/__init__.py (54ms)

And it chews up memory pretty fast. I'm honestly not sure what eventually stops it. We might hit some maximum import resolution depth.

Mactarvish commented 6 months ago

Thanks for your helps. Seems the key point is not depth of directory but num of files. In the soft link under my workspace there's more than 50000000 .jpg files and cause pylance crash. After move the soft link out of my workspace and reload vscode, pylance reworks well.