performance issues editing header files on embedded system with SD card

[indirectlylit]

Describe the bug

• Latest MacOS "client" connected to small embedded linux Ubuntu 18.04 remote machine "target" via SSH and inside that Ubuntu 18.04 docker container using an SD card for storage
• VS Code version 1.67.1
• C/C++ extension version 1.10.2 pre-release (due to https://github.com/microsoft/vscode-cpptools/issues/9247)
• Only other extensions installed are cmake and cmake tools

I have a cmake-based build configuration and intellisense is correctly finding all dependencies and providing full and correct reporting of issues related to type hints.

Functionality such as auto-completion, problem-reporting, and auto-formatting are incredibly slow when editing header files. For example, occasionally after saving a file I'll see this for a solid 10s or so even on a simple file:

Based on this comment I tried setting "C_Cpp.intelliSenseCacheSize": 0 and it has helped the situation.

Steps to reproduce

Set up a C++ project on an embedded system like a Raspberry Pi or Jetson Nano and try editing header files.

Expected behavior

Relatively consistent and hopefully snappy intellisense and auto-formatting performance

Code sample and logs

Configurations in c_cpp_properties.json

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**"
            ],
            "defines": [],
            "compilerPath": "/usr/bin/gcc",
            "cStandard": "c11",
            "cppStandard": "gnu++14",
            "intelliSenseMode": "linux-gcc-arm64",
            "configurationProvider": "ms-vscode.cmake-tools"
        }
    ],
    "version": 4
}

Logs from running C/C++: Log Diagnostics from the VS Code command palette

-------- Diagnostics - 5/20/2022, 5:49:21 PM
Version: 1.10.2
Current Configuration:
{
    "name": "Linux",
    "includePath": [
        "${workspaceFolder}/**"
    ],
    "defines": [],
    "compilerPath": "/usr/bin/gcc",
    "cStandard": "c11",
    "cppStandard": "gnu++14",
    "intelliSenseMode": "linux-gcc-arm64",
    "configurationProvider": "ms-vscode.cmake-tools",
    "compilerPathIsExplicit": true,
    "cStandardIsExplicit": true,
    "cppStandardIsExplicit": true,
    "intelliSenseModeIsExplicit": true,
    "mergeConfigurations": false,
    "browse": {
        "path": [
            "${workspaceFolder}/**"
        ],
        "limitSymbolsToIncludedHeaders": true
    }
}
Custom browse configuration: 
{
    "browsePath": [
        "##redacted##",
        "##redacted##",
        "##redacted##",
        "##redacted##",
        "##redacted##"
    ],
    "compilerPath": "/usr/bin/aarch64-linux-gnu-g++",
    "compilerArgs": [
        "-g"
    ],
    "compilerArgsLegacy": [
        "-g"
    ]
}
Custom configurations:
[ ##redacted## ]
{
    "defines": [
        "GLFW_DLL"
    ],
    "includePath": [
        "##redacted##",
        "##redacted##",
        "##redacted##",
        "##redacted##",
        "##redacted##"
    ],
    "compilerPath": "/usr/bin/aarch64-linux-gnu-g++",
    "compilerArgs": [
        "-g"
    ]
}
Translation Unit Mappings:
[ ##redacted## ]:
    ##redacted## *
Translation Unit Configurations:
[ ##redacted## ]:
    Process ID: 4899
    Memory Usage: 200 MB
    Compiler Path: /usr/bin/aarch64-linux-gnu-g++
    Includes:
        ##redacted##
        ##redacted##
        ##redacted##
        ##redacted##
        ##redacted##
        ##redacted##
    Defines:
        GLFW_DLL
    Standard Version: c++14
    IntelliSense Mode: linux-gcc-arm64
    Other Flags:
        --g++
        --gnu_version=70500
Total Memory Usage: 200 MB

------- Workspace parsing diagnostics -------
Number of files discovered (not excluded): 29046
Number of files parsed: 2771

Logs from the language server logging - let me know if this would be helpful.

Additional context

Let me know if there are further steps I can easily take to identify offending processes. I briefly looked at the Troubleshooting page, but the perf profiling tool does not seem to be available on the embedded ARM platform I'm currently developing against (NVIDIA tegra) and compiling/installing it didn't go smoothly.

[Colengms]

Hi @indirectlylit . Thanks for reporting this. We can use this issue to track investigating further. However, given the variety of different features that exhibit performance issues (i.e. Formatting, which does not leverage our IntelliSense engine), I suspect the performance issue may be related to SSH or Docker. If so, you may be able to confirm this if you detect similar issues with other extensions, such as language services for other languages.

[indirectlylit]

Thanks for response. I understand that this is a tricky one because I can't give clear steps to reproduce.

On my side, I can say that:

• neither client or target device are running any resource-intensive processes
• the two devices are on the same local network and no other applications have shown indication of slow connections between them
• all other resource-intensive applications I've run in docker container have been very fast. The NVIDIA L4T containers mostly just delegate to their host

you may be able to confirm this if you detect similar issues with other extensions, such as language services for other languages

If you have any suggestions on a test I can run to narrow down where the problem is I'd be happy to help. Otherwise I'd recommend we close this issue so it doesn't become a catch-all for a range of performance issues.

[indirectlylit]

(I'd also be interested in better understanding the implications of setting "C_Cpp.intelliSenseCacheSize": 0)

[sean-mcmanus]

The formatting is likely just getting stuck being an IntelliSense update operation.

The intelliSenseCache setting will cause .ipch files to be created/read which store info about the headers -- on some setups this may hurt performance, particular if the headers are being modified.

[indirectlylit]

Hi, thank you @sean-mcmanus

The formatting is likely just getting stuck being an IntelliSense update operation

Do you mean that it might be blocked by or stuck behind an IntelliSense update?

The intelliSenseCache setting will cause .ipch files to be created/read which store info about the headers -- on some setups this may hurt performance, particular if the headers are being modified.

Yes, I think this is indeed the culprit.

default behavior	setting "C_Cpp.intelliSenseCacheSize": 0
> 300MB main.ipch file under /root/.cache/vscode-cpptools/ipch	empty ipch dir
editing a .hpp file causes "Updating IntelliSense" to run for > 30s	editing a .hpp file causes "Updating IntelliSense" to run for just a few seconds

On the other hand, editing .cpp files is fast in both cases. The "intermittency" was just whether I was editing headers or not. I'll update the title.

[sean-mcmanus]

I mean the thread that handles the formatting message is stuck in a queue waiting for the IntelliSense update to finish.

A related issue is https://github.com/microsoft/vscode-cpptools/issues/5362 -- we should probably disable the ipch cache for the TU automatically if the header is editted instead of repeatedly rebuiding it.

[indirectlylit]

Thanks, that makes sense.

On embedded systems like the Raspberry Pi or Jetson Nano with slow SD cards used for storage, reading or writing large amounts of data can be a major bottleneck. In my situation, VS Code was essentially unusable until disabling the cache.