vscode-extensions: update_installed_exts.sh generates invalid hashes

kulabun commented 1 year ago

Describe the bug

update_installed_exts.sh is a script for VSCode extensions update. It works great except of the fact that it generates incorrect hashes. Script downloads a version with tag latest, extract its version and calculate its hash. When nix attempts to download it then, it downloads not the latest tag, but a tag based on the extracted version. They are not guaranteed to be the same.

Steps To Reproduce

Steps to reproduce the behavior:

Run update_installed_exts.sh

> wget  https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/applications/editors/vscode/extensions/update_installed_exts.sh
> ./update_installed_exts.sh
...
{
name = "terraform";
publisher = "hashicorp";
version = "2.24.3";
sha256 = "0mrsbpwyq120d402yi5zxz16xnca2wjq6ya7csvmaxs833vv41jc";
}
...

Now download both the latest version and 2.24.3.

> wget https://hashicorp.gallery.vsassets.io/_apis/public/gallery/publisher/hashicorp/extension/terraform/latest/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage -O latest
> wget https://hashicorp.gallery.vsassets.io/_apis/public/gallery/publisher/hashicorp/extension/terraform/2.24.3/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage -O 2-24-3

Both of them have different hashes.

> nix-hash --flat --base32 --type sha256 latest 2-24-3  
0mrsbpwyq120d402yi5zxz16xnca2wjq6ya7csvmaxs833vv41jc
0rcx5cfbm2xq8sarxvh01n3jcy0jr10d2ahpgk0jinjczflpcysj

When I try to rebuild my environment, I predictably get an error:

error: hash mismatch in fixed-output derivation '/nix/store/mdis1yca274r1mz8zma84svgaisriqrf-hashicorp-terraform.zip.drv':
     specified: sha256-TAay9xhId1W3Zkd5gyUXitluwu+/RC8AaUAE7PldOlc=
        got:    sha256-Unt2qftM2ijBfBcq0UDIEngmhw0A7p6VRriLuhwrnWU=

I tried to fix this hash and immediately get the same problem for two other packages packages.

Expected behavior

The script have to generate a hash for the same version it provides in outputs.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Notify maintainers

@AndersonTorres

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/bzkj7bl9ii8zxsf8n08ra5vrqa4zja62-nix-info
copying path '/nix/store/bzkj7bl9ii8zxsf8n08ra5vrqa4zja62-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.74, NixOS, 22.05 (Quokka), 22.05.20221023.471d921`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - channels(konstantin): `"home-manager-22.05.tar.gz"`
 - channels(root): `"nixos-22.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

AndersonTorres commented 1 year ago

I am not the maintainer of this script.

The only thing I did was move it to a more suitable directory.

kulabun commented 1 year ago

I found that some packages are platform specific and some are not. Nix adds arch suffix in the end, but update_installed_exts.sh don't.

This one don't have platform specific versions: https://marketplace.visualstudio.com/items?itemName=asvetliakov.vscode-neovim#version-history

This one have platform specific versions: https://marketplace.visualstudio.com/items?itemName=hashicorp.terraform#version-history

kulabun commented 1 year ago

Looks like it's not a bug in a script. The url used just return non-consistent results. I guess if I just keep retrying updates, eventually all the updates get the same versions they received during hash generation.

> for i in $(seq 10); do 
  wget https://rust-lang.gallery.vsassets.io/_apis/public/gallery/publisher/rust-lang/extension/rust-analyzer/0.4.1260/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
done

> nix-hash --flat --base32 --type sha256 Microsoft.VisualStudio.Services.VSIXPackage*                                                                                    
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
0qb9pbb6vx062yh578pkgbga2qkm3h5799n22fjkzwmkxfvrx977
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
0qb9pbb6vx062yh578pkgbga2qkm3h5799n22fjkzwmkxfvrx977
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
0qb9pbb6vx062yh578pkgbga2qkm3h5799n22fjkzwmkxfvrx977
166i5j2qgnhnci6ngwlh7ry4vs138b0xaa7rvgix2xhh962656yj
0qb9pbb6vx062yh578pkgbga2qkm3h5799n22fjkzwmkxfvrx977

Though, if I use a URL extracted from a website, the result is expected.

> for i in $(seq 10); 
  do wget https://rust-lang.gallerycdn.vsassets.io/extensions/rust-lang/rust-analyzer/0.4.1260/1666745150870/Microsoft.VisualStudio.Services.VSIXPackage                         
done

> nix-hash --flat --base32 --type sha256 Microsoft.VisualStudio.Services.VSIXPackage*                                                                                    
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw
0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw

The URL I used in the first experiment is the one I received from derivation /nix/store/hv4gg0v6l311a701zigbawim0fgvvc2d-rust-lang-rust-analyzer.zip.drv.

warning: Git tree '/home/konstantin/nixconf' is dirty
building the system configuration...
warning: Git tree '/home/konstantin/nixconf' is dirty
error: hash mismatch in fixed-output derivation '/nix/store/hv4gg0v6l311a701zigbawim0fgvvc2d-rust-lang-rust-analyzer.zip.drv':
         specified: sha256-Qsgq8Cpc6pTlXFnJtxxqTgjjNS4dmAXyP9wXSMKEYTM=
            got:    sha256-0ptihEkQdtHj2/ko1cFCI+hNfD6Q8mdNZBbah4Us0Zg=
error: 1 dependencies of derivation '/nix/store/727mbyi04qbz1r3d9w64gbv9qi5m33ka-vscode-extension-rust-lang-rust-analyzer-0.4.1260.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ynxi9rldm5klbkz966ly68x1w1l4x9rk-vscode-extensions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/knm3bfv6i078kf35jhy7vm1sch9l4ar7-vscode-with-extensions-1.69.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/k111khii632xdah9psg6yrjwsffrr54k-home-manager-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/75yqndz153zmz2j4z5i519gg2h357yyp-vscode-with-extensions-1.69.2-fish-completions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4spf4pwa0m9f5l7brhiwdxj3qmm162xl-home-manager-generation.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bpjf7lp1z438ldny4444bm5hbg6gf8f6-user-environment.drv' failed to build
error: 1 dependencies of derivation '/nix/store/48v2i7dcwx244ayscb0vaygxcs5x838a-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/nv8pqcgak76gfjdf27qzf77dv62q4n5n-nixos-system-konstantin-desktop-22.05.20221023.471d921.drv' failed to build

> cat /nix/store/hv4gg0v6l311a701zigbawim0fgvvc2d-rust-lang-rust-analyzer.zip.drv          
Derive([("out","/nix/store/fjy6wb7pzxawy5grn3id767cnh1dmiis-rust-lang-rust-analyzer.zip","sha256","42c82af02a5cea94e55c59c9b71c6a4e08e3352e1d9805f23fdc1748c2846133")],[("/nix/store/25v14drl573yalys9kdc4c1m8kz5fmvd-bash-5.1-p16.drv",["out"]),("/nix/store/8m2x8d26gx9ghjfb6jd7zz30c6wclq40-curl-7.83.1.drv",["dev"]),("/nix/store/xzmr17csmjpmf2p0m2rp6zd8wdd4dxlz-mirrors-list.drv",["out"]),("/nix/store/ykv69zk4y8dj1ajxrfh2xf17gfqn294x-stdenv-linux.drv",["out"])],["/nix/store/720ikgx7yaapyb8hvi8lkicjqwzcx3xr-builder.sh"],"x86_64-linux","/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16/bin/bash",["-e","/nix/store/720ikgx7yaapyb8hvi8lkicjqwzcx3xr-builder.sh"],[("SSL_CERT_FILE","/no-cert-file.crt"),("buildInputs",""),("builder","/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16/bin/bash"),("configureFlags",""),("curlOpts",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck",""),("doInstallCheck",""),("downloadToTemp",""),("executable",""),("impureEnvVars","http_proxy https_proxy ftp_proxy all_proxy no_proxy NIX_CURL_FLAGS NIX_HASHED_MIRRORS NIX_CONNECT_TIMEOUT NIX_MIRRORS_alsa NIX_MIRRORS_apache NIX_MIRRORS_bioc NIX_MIRRORS_bitlbee NIX_MIRRORS_centos NIX_MIRRORS_cpan NIX_MIRRORS_debian NIX_MIRRORS_fedora NIX_MIRRORS_gcc NIX_MIRRORS_gentoo NIX_MIRRORS_gnome NIX_MIRRORS_gnu NIX_MIRRORS_gnupg NIX_MIRRORS_hackage NIX_MIRRORS_hashedMirrors NIX_MIRRORS_ibiblioPubLinux NIX_MIRRORS_imagemagick NIX_MIRRORS_kde NIX_MIRRORS_kernel NIX_MIRRORS_luarocks NIX_MIRRORS_maven NIX_MIRRORS_mozilla NIX_MIRRORS_mysql NIX_MIRRORS_openbsd NIX_MIRRORS_opensuse NIX_MIRRORS_osdn NIX_MIRRORS_postgresql NIX_MIRRORS_pypi NIX_MIRRORS_qt NIX_MIRRORS_roy NIX_MIRRORS_sageupstream NIX_MIRRORS_samba NIX_MIRRORS_savannah NIX_MIRRORS_sourceforge NIX_MIRRORS_steamrt NIX_MIRRORS_tcsh NIX_MIRRORS_testpypi NIX_MIRRORS_ubuntu NIX_MIRRORS_xfce NIX_MIRRORS_xorg"),("mirrorsFile","/nix/store/kpy64rkpq32sc9c39lc8yqwcb5gdyx22-mirrors-list"),("name","rust-lang-rust-analyzer.zip"),("nativeBuildInputs","/nix/store/6ddzfbih16zb596rpkg9aw2yhnkhn6m5-curl-7.83.1-dev"),("nixpkgsVersion","22.05"),("out","/nix/store/fjy6wb7pzxawy5grn3id767cnh1dmiis-rust-lang-rust-analyzer.zip"),("outputHash","0cv1hk14h5yw7zr0b60x5qsy622fd8fbgjarbkjr9sjw5bq2mj22"),("outputHashAlgo","sha256"),("outputHashMode","flat"),("outputs","out"),("patches",""),("postFetch",""),("preferHashedMirrors","1"),("preferLocalBuild","1"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("showURLs",""),("stdenv","/nix/store/cckh9b9b92zb3a6jincm8aqdm4bff73l-stdenv-linux"),("strictDeps",""),("system","x86_64-linux"),("urls","https://rust-lang.gallery.vsassets.io/_apis/public/gallery/publisher/rust-lang/extension/rust-analyzer/0.4.1260/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage")])

kulabun commented 1 year ago

I guess @jraygauthier is a maintainer.

kulabun commented 1 year ago

A solved a problem for myself by changing a format extensions.nix file use. I think fixing URL is a right way to do it as fetching it by version only doesn't guarantee consistent result.

    {
      name = "rust-lang-rust-analyzer-0.4.1260";
      src = {
        url = "https://rust-lang.gallerycdn.vsassets.io/extensions/rust-lang/rust-analyzer/0.4.1260/1666745150870/Microsoft.VisualStudio.Services.VSIXPackage";
        sha256 = "0c62ww08ayl6k9zg73lgvs8kzgp0vbq5zinplkbnsmvpmv04xvpw";
        name = "rust-analyzer-rust-lang.zip";
      };
      vscodeExtUniqueId = "rust-lang.rust-analyzer";
    }

Here is how I get the URLs. For extensions that require to provide architecture, I use a hard-coded one. I am not sure how architecture is resolved for the link used in original script and nixpkgs. I will be happy to create MR, but want to get some feedback on my findings and format change(it can prefer URL and fallback to current syntax) proposal before I jump into that.

icyrockcom commented 1 year ago

@kulabun I had the same issue with the same extension (rust-analyzer). Using your updates:

https://github.com/kulabun/nixconf/blob/6918f0000a6dc6af8e9a7485de29ee935efeaa33/modules/home-manager/vscode/update-extensions.sh#L46

https://github.com/kulabun/nixconf/blob/6918f0000a6dc6af8e9a7485de29ee935efeaa33/modules/home-manager/vscode/default.nix#L21

worked for me. Obviously I'm on linux-x64.

tricktron commented 1 year ago

@kulabun @icyrockcom ~~I also ran into this issue. Instead of finding the latest version from the url first, I use the version from the unzipped extension and just replace latest with that version:~~

~~https://github.com/tricktron/my-system-configs/blob/d7722c236668cdffab91883c8bd6a2d72001d125/apps/default.nix#L42~~

~~The architecture is not considered at all but so far it is working.~~

Edit: I still got non-determinstic hashes with the above approach. So I also used the solution from @kulabun to get the exact url: https://github.com/tricktron/my-system-configs/blob/9227620913b8ecee917ee4624a685cbd5c4a3c5c/apps/default.nix#L51-L57

jkachmar commented 1 year ago

I wrote a small script to help managing this stuff in a JSON blob w/ a supporting Nix expression; I'm sure there are some improvements to be made, but I mainly wanted the ability to share among different platforms without too much frustration.

https://github.com/jkachmar/morioh/blob/01d4d7461b2a1af4c869ec3e1dff18ce8ff916be/config/user/vscode/update.py

Usage example: https://github.com/jkachmar/morioh/blob/main/config/user/vscode/extensions.nix

NixOS / nixpkgs