Closed spikespaz closed 1 year ago
Hi, @spikespaz !
The crawler uses nix store prefetch-file extension_url (see this line). You may see the extension_url
in the generated nix files. E.g., in data/generated/vscode-marketplace.nix
BTW, I checked with the latest version and it worked
nix-repl> :lf github:nix-community/nix-vscode-extensions/56efe16c813eef6328812856a704a41af7645925
Added 14 variables.
nix-repl> :b extensions.x86_64-linux.vscode.rust-lang.rust-analyzer
This derivation produced the following outputs:
out -> /nix/store/alawnirlr33pl73yrylacpf6x7cyvvh2-vscode-extension-rust-lang-rust-analyzer-0.4.1385
We had this problem before. It seems that the API is not always deterministic. I was able to get different hashes for the same version of the same extension on the same machine after trying to request it enough times.
@AmeerTaweel Can the generator be adjusted to verify the hashes to prevent this mistake from affecting the end-user? I don't think it is a great thing to require the user to write correcting overlays for the faulty fetchers.
@spikespaz the problem is not with the fetcher. The API does not always provide the same file. So even if you write an overlay with the hash you are getting now, you might still get a file with another hash later.
And in the same way, even if the generator verified the hash and it was correct, you might still get a wrong hash later.
As mentioned in the issue in the original repo, we used curl
to fetch files from the API and checked the hashes manually. So the issue is with the API itself.
Curious, I've been using vscode with nix for years and had my own list of extensions that I maintained and used with vscode-utils.extensionFromVscodeMarketplace
. Now I decided to switch to using nix-vscode-extensions
and I get mismatching hash the second day.
I guess I was just lucky before?
I also noticed some problems when I tried to fetch redhat-java via nix from the VSCode marketplace some time ago. In my case, I sometimes got the Linux build of the extension, sometimes I got the OSX build when using the exact same URL. That made me think that maybe only those extensions are affected that have different assets for different architectures, and there's some query parameter or HTTP header involved that needs to be set? E.g. to select the architecture? The absence of that parameter could mean "don't care"... (In my case case, I've opted to use the assets from the GitHub releases page instead of the marketplace.)
I also noticed some problems when I tried to fetch redhat-java via nix from the VSCode marketplace some time ago. In my case, I sometimes got the Linux build of the extension, sometimes I got the OSX build when using the exact same URL. That made me think that maybe only those extensions are affected that have different assets for different architectures, and there's some query parameter or HTTP header involved that needs to be set? E.g. to select the architecture? The absence of that parameter could mean "don't care"... (In my case case, I've opted to use the assets from the GitHub releases page instead of the marketplace.)
Could that be targetPlatform
query param used by vscode-utils.extensionFromVscodeMarketplace
?
That would explain why vscode-utils.extensionFromVscodeMarketplace
never caused hash mismatch for me in the past.
I did some experiments and it seems true.
When I run the following:
$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage" -o extension.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1108k 100 1108k 0 0 1402k 0 --:--:-- --:--:-- --:--:-- 1403k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-Q1R+2t/tWI3haoF5fW7k2413BLIMf1ImM4J9CUrtjnQ=
Then I tried different types of architectures, and this is the one that matched:
$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage?targetPlatform=alpine-arm64" -o extension.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1108k 100 1108k 0 0 1319k 0 --:--:-- --:--:-- --:--:-- 1319k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-Q1R+2t/tWI3haoF5fW7k2413BLIMf1ImM4J9CUrtjnQ=
So I was getting the alpine-arm64
version by default. However, I can obtain different versions at will now:
$ curl "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/vscode-yaml/1.10.20220805/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage?targetPlatform=win32-arm64" -o extension.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1108k 100 1108k 0 0 1042k 0 0:00:01 0:00:01 --:--:-- 1043k
$ nix hash to-sri --type sha256 $(nix-hash --flat --base32 --type sha256 extension.zip)
sha256-1aEyB3BTe8S0OGpk3JJLi6Zq6gQLPacOk7Om5suYTr8=
Interestingly, when I ask for the darwin-x64
version or the darwin-arm64
version, I get an error saying that there is no support for the requested target platform.
So we need to put some thought into this to fix our fetcher:
Also, here is a list of supported target platforms if it helps anyone.
The JSON API to find out version/targetPlatform combinations:
cat <<'EOF' | curl -sSL -H 'Accept: application/json;api-version=6.1-preview.1' -H 'Content-Type: application/json' https://marketplace.visualstudio.com/_apis/public/gallery/extensionquery -d @- | jq --raw-output '.results[0].extensions[0].versions[] | .version + " " + .targetPlatform'
{
"assetTypes": [],
"filters": [
{
"criteria": [
{
"filterType": 8,
"value": "Microsoft.VisualStudio.Code"
},
{
"filterType": 7,
"value": "redhat.vscode-yaml"
},
{
"filterType": 12,
"value": "4096"
}
],
"pageNumber": 1,
"pageSize": 2
}
],
"flags": 946
}
EOF
yields
1.11.10112022
1.10.20220805 alpine-x64
1.10.20220805 alpine-arm64
1.10.20220805 linux-armhf
1.10.20220805 win32-arm64
1.10.20220805 win32-ia32
@twz123 thanks for your help!
Correct me if I'm wrong. Are they using alpine
and linux
interchangeably?
I'd guess that alpine ships (probably statically linked) musl binaries, whereas Linux ships usual glibc binaries.
Any news on getting that extra query information working so the hashes can match?
Hello, I am facing this issue today, on rust-lang.rust-analyzer
as well. Other extensions have the proper hash.
Hmm, this works for me
nix-repl> :lf github:nix-community/nix-vscode-extensions/d5a069fccf424791a45722805f5d99cbbe1ae264
Added 15 variables.
nix-repl> :b extensions.x86_64-linux.vscode-marketplace.rust-lang.rust-analyzer
This derivation produced the following outputs:
out -> /nix/store/63ihxcwcaj7s4wra8z27wwai55r5cgd8-vscode-extension-rust-lang-rust-analyzer-0.4.1505
@deemp Because it's random.
Indeed, I’m now able to build rust-lang.rust-analyzer
without updating nix-vscode-extensions
. This is annoying that the VSCode Marketplaces has a non-deterministic API. Has it been reported upstream?
I suggest we continue this discussion in https://github.com/nix-community/nix-vscode-extensions/issues/20
Not sure why, but
rust-lang.rust-analyzer
fails with a bad hash.I would like to know why the hash is wrong in this repository, I don't know how the automation/crawler works, but is it even possible for the hash to change on the VSCode Marketplace?