Workaround for machines that do not have `glibc` >= 2.28

[wlovo]

Problem

Closely related to https://github.com/microsoft/vscode/issues/201129, there will be a significant number of machines who will be unable to utilize the latest version of Visual Studio Code (VS Code) once v1.86 is made publicly available.

It will make all Amazon Linux 2 (AL2) EC2 machines unable to use VS Code, while also affecting enterprise users who may use customized or parallel versions of the glibc library for their own development needs.

While I acknowledge the need to update components to suit the development needs of the VS Code team, it seems a bit heavy-handed to completely block any alternative paths for those who have a customized or advanced need.

There previously was a workaround for those who relied on such needs, but it seems the new script introduced to check requirements is rendering them useless.

Solution

I propose the introduction of a mechanism that allows developers to override the checks for glibc and libstdc++ versions that must be explicitly requested. This could take on the form of a system variable that either disables the check or provides an alternate path to the requested libraries (e.g. VSCODE_GLIBC=/path/to/customized/library), but may not be the only solution available.

[rexm]

Hey team, just checking in to add context to Will's request above that this is going to hard-block many thousands of developers currently using VS Code. A number of leaders at Amazon are tracking this issue, we are very concerned about the impact to those customers and hopeful we can work with you on a workaround like Will suggested.

[deepak1556]

We have documented the proper workaround at https://code.visualstudio.com/docs/remote/faq#_can-i-run-vs-code-server-on-older-linux-distributions which is also included in the error message from the requirements check.

There previously was a workaround for those who relied on such needs, but it seems the new script introduced to check requirements is rendering them useless.

If the alternate glibc is in the ld cache then the check should pick it up, can you please share simple repro steps I can use to repro the issue. It is possible to fix the check if there is a bug in the detection.

[ShawnCarnell]

Thank you, @deepak1556. We might need some help in understanding how this would work in practice.

If we have a system that uses an older glibc as the default, how can we put a newer glibc in place such that only VS Code and its extensions use the new glibc and every other binary on the host continues to use the old glibc (for the sake of system stability and fidelity with production platforms running the older glibc)?

[bookest]

We have documented the proper workaround at https://code.visualstudio.com/docs/remote/faq#_can-i-run-vs-code-server-on-older-linux-distributions which is also included in the error message from the requirements check.

Does this mean 1.85 will continue to receive security fixes or is it "as is" going forward?

My concern here is leaving a significant developer population relying on an unsupported VS Code release. That is essentially a time bomb waiting for security to tell them they have to change up how they work because of a new security issue that won't be patched in 1.85. So ideally I can keep folks on the latest supported VS Code, even if it means I have to own the duct tape and baling wire to make it work on my org's older distro.

Admittedly there's an element of needing update the distro in use, but that's not something that can change quickly in a large deployment.

If the alternate glibc is in the ld cache then the check should pick it up

AFAIK the ld cache is global, so this would effectively mean the entire system has been migrated to an updated glibc, which is not a viable option for my use case (I would just update if it was).

can you please share simple repro steps I can use to repro the issue. It is possible to fix the check if there is a bug in the detection.

I built glibc-2.28 from source and install it in a secondary location (/opt/compat/glibc-2.28). Then used a similar approach as https://github.com/nix-community/nixos-vscode-server to patch the binaries to use the interpreter and RPATH to use the glibc-2.28 install (and fall back to /usr/lib64 for other dynamic libraries). I also setup a systemd path unit to detect when new remote binaries are installed and automatically patch them so remote development would just work when new versions of the server are uploaded.

This approach was working, but then the dependency on libstdc++.so.6 was bumped as well. Touching /etc/NIXOS on my system got me passed that, but then the check-requirements-linux.sh script was introduced, and was back to failing due to outdated dependencies.

At this point, I could continue to build additional updated dependencies and repeat the binary patching hacking around, but I really don't want to get onto a treadmill with both the requirements checks in the CLI and in the check-requirements-linux.sh script. It would be nice if there was a way to opt-out of the requirements check at the system level that didn't require masquerading as NixOS (both via /etc/NIXOS and modifying /etc/os-release (with acknowledgment that the system now owns any bugs from running in this configuration). Maybe something like /etc/vscode-skip-requirements-check-unsupported that disables checks in the same way they are for NixOS.

Does that sound reasonable? Are there any other things I need to be concerned about when taking the patching approach? Like does the CLI do any checksum verification when unpacking the binaries that might rollback my changes?

Do you have any other suggestions on how to approach this?

I'm happy to make any contributions necessary to support my use case, but would like to get some agreement on the right approach before digging in.

[isidorn]

Hi VS Code PM here 👋

From my understanding you all work for Amazon @wlovo, @rexm @ShawnCarnell @bookest. If there is someone here not from Amazon then can you please share why this issue is important for your use case - thank you 🙏

Amazon folk - can you please send me an email to inikolic@microsoft.com and we are happy to meet with you next week to understand the blockers you hit while trying to do the workaround @deepak1556 provided https://code.visualstudio.com/docs/remote/faq#_can-i-run-vs-code-server-on-older-linux-distributions

[promitbiswas]

Thanks @isidorn. We've reached out over email. Looking forward to connect next week. In the meantime, do you have a planned release date for v1.86 you can share with us?

[isidorn]

Thank you 🙏
Expect response to email on Monday early morning since we are in Europe and folk are out for the weekend. Planned release data 31. January.

[theswit]

My org (not amazon) is on a RHEL7 compatible OS for the time being. It's EOL'd later this year, but not for several months if i understand correctly. I'm sure there are many people out there in a similar situation.

This thread and the other bugs make it sound like this is not going to be fixed, so people using EL7 will be stranded on 1.85 until they switch OSs (not an easy or quick thing to do), sounds like. EL7 can't build glibc 2.28 from source (needs an updated gcc as well, and i'm sure many other dependencies will need updating). It'd be good to have some path forward that doesn't require big teams to switch OS's.

[albireox]

My team is in the same situation. I work in academia and my team and I develop software for operations in an astronomical observatory. All our servers are running on CentOS 7 and while we plan to update some of the external-facing gateways after EOL, I don't expect that we'll update most machines until the project is over in a couple years. My team heavily uses VSCode to develop directly on those servers and there won't be a good way for us to do this after the 1.86 release.

I appreciate the need to drop support in certain situations, but I it would be great to have a solution that allows these old system to continue working even if it's with limited support or new features.

[joaomoreno]

@albireox this is fascinating, I love seeing all the different places VS Code is used, thanks for sharing!

To add one workaround to the mix: if Docker can be installed in the remote machines, dev containers via SSH is a fully supported scenario. @theswit, @albireox have you considered this before?

[joaomoreno]

@deepak1556 reminded me that still in this scenario the remote SSH agent needs to run, making it not really a viable workaround.

[albireox]

@albireox this is fascinating, I love seeing all the different places VS Code is used, thanks for sharing!

Thanks for developing VSCode! VSCode, especially the SSH extension, has been a real game-changer in terms of our workflow, as we often have to edit code on the fly in multiple servers and for many software products.

To add one workaround to the mix: if Docker can be installed in the remote machines, dev containers via SSH is a fully supported scenario. @theswit, @albireox have you considered this before?

I considered it, but as you say I think that won't work. Our current alternative, which is not great and depends on IT being willing to do this for us, is to spin a VM more modern OS in each server and then cross-mount some of the resources that we edit with VSCode. This is not a full solution because for example most of our products are Python packages that run in virtual environments. The virtual environments won't work in the new OS since they the linked libraries are not in the same locations. So we would need to create new virtual environments for each Python product also in the new OS and try to not get too confused about what we are running where :)

[gakio12]

I'd also like to state our use case. We have various OS environments we must develop and being in that environment directly speeds up development significantly. They range from CentOS 5 all the way to Debian Bookworm. Currently we are unable to use VSCode remote-server in CentOS 5 and CentOS 6 environments, and losing CentOS 7 will be big loss as it creates a fragmented development process between the various systems we support (it isn't feasible to upgrade all the OS's as they are part of long life equipment).

However, if vscode-server was installed either as a static binary, installed with all of it's required libraries in a virtual environment, or had the ability to determine where to find its required glibc libraries via an environment variable or config, that would not only ensure we are able to continue doing remote development in a CentOS 7 environment, but it would also allow us to open up CentOS 6 and CentOS 5. This allows us to unify our development teams' process around VSCode and the Remote SSH extension.

If vscode-server was installed on a remote machine bundled with it's required libraries, that would reduce the dependencies on the remote system's OS and make a glibc change no longer mark an entire remote OS as unsupported.

[wojtasemzi]

Similar as @gakio12 CentOS 7 in synchrotron, and no view to update in near future...

[Elastino]

@isidorn First of all, thank you very much for the greatest product for developers. Secondly, as since my primary language isn't English, this is about informational not blaming. (Personally, I really love what you are doing.) Thirdly, I'm not affiliated with Amazon at all but more than thousands my colleagues are getting bumped in.

Here goes why.

• In case of huge project, we can't simply get away from old OS, and go for upgrading host OS. (e.g. mostly dependency to Python version in our case)
• Not all people don't read all logs. (e.g. VS Code did print out it failed to launch due to minimum requirement however this is printed(=logged) in one line within thousands lines of logs because "Waiting for server log..." prints out every 8ms, and there is NO explanation why it failed. Unfortunately, VS code GUI offers options of "Retry","Close" and "Do something else" options. So far, none of them really tell the user WHY/WHAT we are failing.)
• About the warning which was noticed in Summer 2023, probably, including me, it was very difficult understand what would be the consequence of having this GLIBC minimum requirement changes. Rather than stating GLIBC minimum version, it would be great if someone stated major linux-distros including Ubuntu 18 and etc. In that case, we would have more time to investigate how to achieve side-loading of GLIBC on older system.

Finally, I hope we can put some technical workaround here all together so we can mitigate this impact ASAP.

Best regards, CJ Lee

[simozacca]

Dear developpers,

Thank you so much for your work; I lead a team of cancer researchers and part of a consortium of ~100 people with many proudly using now VSCode for data analysis and software development. Our academic systems (HPC of research centers hosting thousands of people) do not satisfy the new minimum requirements and this new update has become absolutely disruptive for us. I hope that you understand that many HPCs, especially in academic contexts, cannot easily update their systems. We beg to please re-consider this decision or add an alternative workaround, which is different than being left behind.

Thank you

[wynnw]

Here's the workaround I came up with as the update today broke my whole development team (we use amazonlinux2 as well). We've been dealing with various projects moving on past the glibc level of this distro and we build our own node v18 as a result.

On our wsl instances and lima-vm.io vm's we can do:

cd ~/.vscode-server/bin/05047486b6df5eb8d44b2ecd70ea3bdf775fd937
mv node node-vscode
ln -sf /path/to/our/custom/build/of/node .
touch /tmp/vscode-skip-server-requirements-check

this works great with limitations like the terminal doesn't work. Seems like it would be a much more simple workaround for people to just build their own node v18 binary than try to manage an alternate glibc installation which is a nightmare. Being able to npm install the dependencies like node-pty (which has a binary component that can be built on the system) would be really nice too.

[rogeriosouzamoraes]

Hi VS Code PM here 👋

From my understanding you all work for Amazon @wlovo, @rexm @ShawnCarnell @bookest. If there is someone here not from Amazon then can you please share why this issue is important for your use case - thank you 🙏

Amazon folk - can you please send me an email to inikolic@microsoft.com and we are happy to meet with you next week to understand the blockers you hit while trying to do the workaround @deepak1556 provided https://code.visualstudio.com/docs/remote/faq#_can-i-run-vs-code-server-on-older-linux-distributions

I am not from Amazon, the issue affected lots of developers here who still use RHEL 7.x machines. RHEL 7.9 is still supported by Red Hat, you should consider updating this kind of requirement after major Linux vendors ending the support to the OSs.

[virtuald]

Worth noting that Ubuntu 18 is still supported for 5 more years if you pay for security updates.

[nskelley]

If there is someone here not from Amazon then can you please share why this issue is important for your use case - thank you 🙏

Following up here as well. I work in research at MIT, and it looks like our cluster is on CentOS 7. Most research faculty and staff don't have access to/any say in what the servers run, and similar to @simozacca and some others, a system-wide upgrade doesn't seem to be on the immediate horizon---possibly in the late summer when MGHPCC shuts down.

[wynnw]

Here's a funner workaround I've built for myself and my dev teams that are on amazlinux2. So far everything is working for me, including the terminal, run targets, etc. Maybe it can help you out? It requires a locally build node v18 - make sure you change the path for mynode.

#!/bin/bash

# This is a script to make the vscode installation on amazonlinux2 work with a locally built node v18
# by swapping out some binary components for versions built locally
set -e

vscodever="05047486b6df5eb8d44b2ecd70ea3bdf775fd937"
vscodetop="${HOME}/.vscode-server"
vscodedir="${vscodetop}/bin/${vscodever}"
mynode="/path/to/your/node/v18/binary/build/node"

# this dependency is different - it bundles it's own builds, so lets copy it's own package binaries
# If we can't find it, exit out before doing anything
sysarch=$(arch)
if [ "${sysarch}" = "x86_64" ]; then
    watcherpkg="@parcel/watcher@2.1.0"
elif [ "${sysarch}" = "aarch64" ]; then
    watcherpkg="@parcel/watcher-linux-arm64-glibc@2.2.0"
else
    echo "Unsupported architecture for @parcel/watcher"
    exit 1
fi

# point vscode to our node build
echo "Using system node v18"
cd ${vscodedir}
ln -sf ${mynode} .
touch /tmp/vscode-skip-server-requirements-check

# rebuild vscode npm dependencies with binary builds. The list was generated by
# doing a npm ls and then looking for all installs that had a build/Release directory
echo "Building npm binary dependencies"
mkdir -p ~/.vscode-localbuilds
cd ~/.vscode-localbuilds

export PATH=$(dirname ${mynode}):$PATH

npm install --no-save \
    kerberos@2.0.1 \
    native-watchdog@1.4.2 \
    node-pty@1.1.0-beta5 \
    ${watcherpkg} \
    @vscode/spdlog@0.14.0

cp node_modules/kerberos/build/Release/kerberos.node \
    ${vscodedir}/node_modules/kerberos/build/Release/
cp node_modules/native-watchdog/build/Release/watchdog.node \
    ${vscodedir}/node_modules/native-watchdog/build/Release/
cp node_modules/node-pty/build/Release/pty.node \
    ${vscodedir}/node_modules/node-pty/build/Release/
cp node_modules/@vscode/spdlog/build/Release/spdlog.node \
    ${vscodedir}/node_modules/@vscode/spdlog/build/Release/

# custom logic for the watcher package
if [ "${sysarch}" = "x86_64" ]; then
    cp node_modules/@parcel/watcher/prebuilds/linux-x64/node.napi.glibc.node \
        ${vscodedir}/node_modules/@parcel/watcher/build/Release/watcher.node
elif [ "${sysarch}" = "aarch64" ]; then
    cp node_modules/@parcel/watcher-linux-arm64-glibc/watcher.node \
        ${vscodedir}/node_modules/@parcel/watcher/build/Release/watcher.node
fi

echo "Complete - restart vscode and connect to your wsl/ssh amazonlinux2 remote"

[Mads-PeterVC]

I'm a postdoc doing computational materials science, I have been using VS Code with remote-SSH for several years and am very appreciative of the efforts of the VS Code team.

However, I am in a similar position as others - my work happens on a university owned cluster that runs CentOS 7. I have very little/no say on the choice of OS on the cluster, so this change is rather unfortunate for me and many of my colleagues. Being able to do my development work directly on the cluster with VS Code has been huge productive boost for me. I do everything from writing new code, submitting jobs to the queueing system, doing analysis in notebooks directly where the output data is stored, to writing papers in LateX on the cluster. I connect to the server in regular terminal perhaps once a month, usually because htop looks better to me there.

I have been (kindly) teasing the emacs/vim users in my department and encouraging them to just try VS Code. Yet unless this issue is resolved in a not excessively technical way I guess they might have been right to stick to their ancient ways.

Really hope a solution can be found and thanks again for all the effort.

[chadhutchins182]

Several of our users at @U-S-NRL-Marine-Meteorology-Division are reporting issues with Version 1.86 that was just released with our RHEL 7 systems.

[mknwebsolutions]

FYI This update has blocked all our development teams across the board. Looking for solution.

[henghamao]

We met the same issue. For now, we disabled the auto update, and reinstalled the old version v.1.8.5. Hope the issue could be solved in future version.

[jamestiotio]

If there is someone here not from Amazon then can you please share why this issue is important for your use case - thank you 🙏

Chipping in as a software engineer at an MNC developing enterprise-grade software, this issue seems to affect a significant number of developers here as well. While migration to newer Linux distro versions has been planned, meticulous and extensive planning is needed especially if it involves a large number of developers distributed globally. Proper infrastructure and internal tooling support are also needed to ensure a smooth transition. As such, it might take a while until we upgrade to the latest Linux distro versions.

The workaround that we are currently using is downgrading VSCode from version 1.86.0 to 1.85.2 and the "Remote - SSH" extension from version 0.108.0 to 0.107.1 and pinning them there (by disabling automatic updates). This is a little concerning since bug fixes, security patches, and new features from the latest versions of VSCode are still very much desirable. Our team (and I'm sure others around the world as well) would definitely appreciate it if a proper solution with official support is implemented.

Thank you folks!

[pfaion]

I also work remotely at a company still using some CentOS 7 machines. VSCode remote stopped working on those unfortunately. However, CentOS 7 is only marked as end-of-life in June 2024, so it caught us off guard to have VSCode drop support before the EOL date!

[zimskyzeng]

me too. so many "Waiting for server log..."

[inhzus]

Here's a funner workaround I've built for myself and my dev teams that are on amazlinux2. So far everything is working for me, including the terminal, run targets, etc. Maybe it can help you out? It requires a locally build node v18 - make sure you change the path for mynode.

#!/bin/bash

# This is a script to make the vscode installation on amazonlinux2 work with a locally built node v18
# by swapping out some binary components for versions built locally
set -e

vscodever="05047486b6df5eb8d44b2ecd70ea3bdf775fd937"
vscodetop="${HOME}/.vscode-server"
vscodedir="${vscodetop}/bin/${vscodever}"
mynode="/path/to/your/node/v18/binary/build/node"

# this dependency is different - it bundles it's own builds, so lets copy it's own package binaries
# If we can't find it, exit out before doing anything
sysarch=$(arch)
if [ "${sysarch}" = "x86_64" ]; then
    watcherpkg="@parcel/watcher@2.1.0"
elif [ "${sysarch}" = "aarch64" ]; then
    watcherpkg="@parcel/watcher-linux-arm64-glibc@2.2.0"
else
    echo "Unsupported architecture for @parcel/watcher"
    exit 1
fi

# point vscode to our node build
echo "Using system node v18"
cd ${vscodedir}
ln -sf ${mynode} .
touch /tmp/vscode-skip-server-requirements-check

# rebuild vscode npm dependencies with binary builds. The list was generated by
# doing a npm ls and then looking for all installs that had a build/Release directory
echo "Building npm binary dependencies"
mkdir -p ~/.vscode-localbuilds
cd ~/.vscode-localbuilds

export PATH=$(dirname ${mynode}):$PATH

npm install --no-save \
    kerberos@2.0.1 \
    native-watchdog@1.4.2 \
    node-pty@1.1.0-beta5 \
    ${watcherpkg} \
    @vscode/spdlog@0.14.0

cp node_modules/kerberos/build/Release/kerberos.node \
    ${vscodedir}/node_modules/kerberos/build/Release/
cp node_modules/native-watchdog/build/Release/watchdog.node \
    ${vscodedir}/node_modules/native-watchdog/build/Release/
cp node_modules/node-pty/build/Release/pty.node \
    ${vscodedir}/node_modules/node-pty/build/Release/
cp node_modules/@vscode/spdlog/build/Release/spdlog.node \
    ${vscodedir}/node_modules/@vscode/spdlog/build/Release/

# custom logic for the watcher package
if [ "${sysarch}" = "x86_64" ]; then
    cp node_modules/@parcel/watcher/prebuilds/linux-x64/node.napi.glibc.node \
        ${vscodedir}/node_modules/@parcel/watcher/build/Release/watcher.node
elif [ "${sysarch}" = "aarch64" ]; then
    cp node_modules/@parcel/watcher-linux-arm64-glibc/watcher.node \
        ${vscodedir}/node_modules/@parcel/watcher/build/Release/watcher.node
fi

echo "Complete - restart vscode and connect to your wsl/ssh amazonlinux2 remote"

This workaround works for me. huge thx for ur sharing!

[guppy42]

"solution" for Linux users install code via flatpak ( it's still on 1.85 as of now ) and then mask the package: flatpak mask com.visualstudio.code

so it doesn't update.

ofc the flatpak version wont have your extension, project history ,etc - but atleast you can connect :/

you can still keep your system package for newer servers

[zimskyzeng]

Thank you，I have received the message~Zimsky Zeng

[joaomoreno]

Locking this issue as well. Please read https://github.com/microsoft/vscode/issues/203967#issuecomment-1923440629.

We'll provide an update as soon as we have a plan to go forward from here.

[isidorn]

Thank you very much for your feedback and thank you for sharing your passion for VS Code and sharing how it is being used to enable various scenarios. We have discussed this more in the VS Code team and we have decided to allow VS Code to connect to an OS that is not supported by VS Code (no support for glibc >= 2.28) for 12 more months.

We hope this will provide the needed time for you and your companies to migrate to newer Linux distributions. VS Code will show the appropriate dialog and banner that you are connecting to an OS that is not supported by VS Code.

We expect these changes to land in our recovery release that will happen later this week. Thank you

[isidorn]

Closing this issue as the recovery release that allows connections to an OS that is not supported by VS Code will be released on Thursday (8 Feb). Please try it out once it is released, and in case you see problems feel free to file new issues.

Thanks again for your feedback.

[isidorn]

Gentle reminder for everyone on this thread. As we announced here https://github.com/microsoft/vscode/issues/203375#issuecomment-1927893504

VS Code will allow to connect to an OS that is not supported by VS Code (no support for glibc >= 2.28) until Feb 2025. Thus this flow will only be supported for 9 more months. Please reach out to your Company admins to update your Linux distributions until then.

Thank you for your understanding 🙏