Support platform-specific extensions

deemp commented 1 year ago

As mentioned in https://github.com/nix-community/nix-vscode-extensions/issues/5, the API is non-deterministic without setting the target platform in a query parameter.

@twz123 proposed a way to collect a version - target platform mapping in https://github.com/nix-community/nix-vscode-extensions/issues/5#issuecomment-1419411697.

We can add this information to our cache.

For each extension (name, publisher), for each platform, we need an object containing the fields lastUpdated, sha, version. We don't need a url because we can construct it on the fly in Nix.

Our .jsons will be 2+ times larger. I guess, we can for now provide extensions just for Nix-supported platforms.

What should be the mapping from Nix-supported platforms to Marketplace target platforms?

Nix supports:

Linux i686
Linux x86_64
Linux aarch64
macOS x86_64
macOS aarch64

VSCode Marketplace supports (excluding win):

universal
linux-x64
linux-arm64
linux-armhf
alpine-x64
alpine-arm64
darwin-x64
darwin-arm64
web

Open VSX supports

universal
linux-x64
linux-arm64
linux-armhf
alpine-x64
alpine-arm64
darwin-x64
darwin-arm64
web

twz123 commented 1 year ago

I don't think that platform specific packages will just work as-is on NixOS. I've looked at rust-analyzer linux-x64:

diff --git a/data/cache/vscode-marketplace.json b/data/cache/vscode-marketplace.json
index 2eb7a7e..8a249b0 100644
--- a/data/cache/vscode-marketplace.json
+++ b/data/cache/vscode-marketplace.json
@@ -34516,7 +34516,7 @@
 , {"lastUpdated":"2017-02-01T00:27:58.623Z","name":"markdown-dir","publisher":"russoturisto","sha256":"sha256-c/YEfJfeG7avzKyHLlOQhlePmZLsjWfWjixf/r0o4pQ=","url":"https://russoturisto.gallery.vsassets.io/_apis/public/gallery/publisher/russoturisto/extension/markdown-dir/0.1.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.1.0"}
 , {"lastUpdated":"2022-11-21T14:49:50.243Z","name":"russ-monokai-extended","publisher":"russpitcher","sha256":"sha256-NuDjssoQMNEbY9g8WfZVeH4VKgLoKiCO3Cr8mJd4VzE=","url":"https://russpitcher.gallery.vsassets.io/_apis/public/gallery/publisher/russpitcher/extension/russ-monokai-extended/1.0.1/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"1.0.1"}
 , {"lastUpdated":"2022-11-15T16:02:36.647Z","name":"rust","publisher":"rust-lang","sha256":"sha256-2uV+s//4fAEuibNHjSJop4pdf9KpDHsICDNWmdjJVSs=","url":"https://rust-lang.gallery.vsassets.io/_apis/public/gallery/publisher/rust-lang/extension/rust/0.7.9/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.7.9"}
-, {"lastUpdated":"2023-05-11T00:44:47.053Z","name":"rust-analyzer","publisher":"rust-lang","sha256":"sha256-U1WRHHUm4Yq2u+V6FJcmOmbUpaJ4O4I9lr6vsYwlQZU=","url":"https://rust-lang.gallery.vsassets.io/_apis/public/gallery/publisher/rust-lang/extension/rust-analyzer/0.4.1509/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.4.1509"}
+, {"lastUpdated":"2023-05-11T00:44:47.053Z","name":"rust-analyzer","publisher":"rust-lang","sha256":"sha256-NdQgIPVEEkSQa5Kg4mesydi5nxD0eF6Evf8GHRyyq2g=","url":"https://rust-lang.gallerycdn.vsassets.io/extensions/rust-lang/rust-analyzer/0.4.1509/1683765593921/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.4.1509"}
 , {"lastUpdated":"2022-06-01T20:06:19.093Z","name":"kahatheme","publisher":"rustamkakhramanov","sha256":"sha256-z+d6vx3HlUf9pEilcZ6xzPDkQsIQaqzmoa5Cfr4cVnw=","url":"https://rustamkakhramanov.gallery.vsassets.io/_apis/public/gallery/publisher/rustamkakhramanov/extension/kahatheme/0.1.4/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.1.4"}
 , {"lastUpdated":"2023-03-27T05:41:30.213Z","name":"rust-analyzer-cn","publisher":"rustcc","sha256":"sha256-3STvSEtC0DmvDwzw3l7NiD8RvpqErKCpyVeGCeTLawE=","url":"https://rustcc.gallery.vsassets.io/_apis/public/gallery/publisher/rustcc/extension/rust-analyzer-cn/0.5.3/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.5.3"}
 , {"lastUpdated":"2022-06-24T14:06:41.493Z","name":"night-blossom","publisher":"rustedturnip","sha256":"sha256-VUDHze5YT4052zKMqdzGW2V2Kc+KeY2Cpth9Si4MTYM=","url":"https://rustedturnip.gallery.vsassets.io/_apis/public/gallery/publisher/rustedturnip/extension/night-blossom/0.1.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage","version":"0.1.0"}

$ nix build .#extensions.x86_64-linux.vscode-marketplace.rust-lang.rust-analyzer
warning: Git tree '/home/twieczorek/Repos/nix-vscode-extensions' is dirty

$ file result/share/vscode/extensions/rust-lang.rust-analyzer/server/rust-analyzer
result/share/vscode/extensions/rust-lang.rust-analyzer/server/rust-analyzer: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=39f93ace56036127c01532346574e52aaacf4b97, with debug_info, not stripped

$ ldd result/share/vscode/extensions/rust-lang.rust-analyzer/server/rust-analyzer
        linux-vdso.so.1 (0x00007ffc4259e000)
        libgcc_s.so.1 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libgcc_s.so.1 (0x00007f375e1dc000)
        librt.so.1 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/librt.so.1 (0x00007f375e1d7000)
        libpthread.so.0 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libpthread.so.0 (0x00007f375e1d2000)
        libm.so.6 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libm.so.6 (0x00007f375e0f2000)
        libdl.so.2 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libdl.so.2 (0x00007f375e0ed000)
        libc.so.6 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libc.so.6 (0x00007f375bff7000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f375e1f8000)

That binary will not work out-of-the box on my system. Either folks need to be instructed to use something like nix-ld, or those extensions need to be massaged (e.g. with patchelf, or by building the extensions from source).

twz123 commented 1 year ago

Oh, and for the mappings:

OSX seems obvious:

macOS x86_64 <=> darwin-x64
macOS aarch64 <=> darwin-arm64

For Linux, I'd choose the glibc versions:

Linux i686 <=> :broken_heart:
Linux x86_64l <=> linux-x64
Linux aarch64 <=> linux-arm64

The musl versions aren't statically linked either (at least rust-analyzer isn't), and with the glibc versions, there's at least nix-ld which may be used to make it work.

Another thing to consider: What about only selecting the extensions that aren't platform specific. I noticed that rust-analyzer is also offered as a version without any binaries (the targetPlatform is simply omitted in the JSON response). That way, folks could bring their own rust-analyzer binary, from nix, or from rustup or from wherever. That works for rust-analyzer. Not sure how this works for other dependencies, and what the expectations are for the users of this flake.

anilanar commented 1 year ago

@twz123 As a NixOS+vscode user for a very long time, I don't remember seeing an extension that doesn't allow specifying a path for whatever binary it was built for/around. It's typical to provide our own binaries to such extensions. So I don't think fixing binaries is responsibility of this repository. That would be very difficult if not impossible.

twz123 commented 1 year ago

So I don't think fixing binaries is responsibility of this repository. That would be very difficult if not impossible.

Sure! So would it make sense to focus on those extension versions that don't specify any target platform instead? I'd assume that those wouldn't contain any native libraries/binaries.

anilanar commented 1 year ago

So would it make sense to focus on those extension versions that don't specify any target platform instead?

I'm not sure if I understood your questions correctly. When I use nix-vscode-extension.rust-analyzer, I'd expect to get the latest version that is released for the platform that is most similar to mine (as you specified above) and I'd always provide my own binary path to it using relevant extension setting/env var.

So would it make sense to focus on those extension versions that don't specify any target platform instead?

I fear some extensions will not have platform-independent versions.

I noticed that rust-analyzer is also offered as a version without any binaries

I wonder if that is a mistake they made when releasing that specific version.

deemp commented 1 year ago

@twz123, @anilanar, @AmeerTaweel, and other.

The PR https://github.com/nix-community/nix-vscode-extensions/pull/21 hopefully solves this issue. Please, check.

nix-community / nix-vscode-extensions

Support platform-specific extensions #20