Artifact hash mismatch with P4est_jll on Julia v1.10

[stillyslalom]

I'm encountering an error when trying to install P4est_jll in Julia v1.10-beta2 (installed via juliaup) on two different Windows 10 machines - the issue is not encountered on v1.9.3:

(jl_tdZRQD) pkg> add P4est_jll
   Resolving package versions...
  Downloaded artifact: P4est
  Downloaded artifact: P4est
ERROR: Unable to automatically download/install artifact 'P4est' from sources listed in 'C:\Users\Alex\.julia\packages\P4est_jll\kfjxs\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767
- https://github.com/JuliaBinaryWrappers/P4est_jll.jl/releases/download/P4est-v2.8.1+2/P4est.v2.8.1.x86_64-w64-mingw32-mpi+microsoftmpi.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767

Troubleshooting on Slack with @sloede and @fredrikekre:

julia> using Tar, CodecZlib, Downloads, SHA

julia> artifact = Downloads.download("https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138")
"C:\\Users\\Alex\\AppData\\Local\\Temp\\jl_glqlGnAL6b"

julia> shasum = bytes2hex(open(SHA.sha256, artifact))
"1abe2cd4e2c31d34c840d3042f94d6b34d85762c6be0fbd480eaf52136b566e1"

julia> open(GzipDecompressorStream, artifact) do io
           tree_hash = Tar.tree_hash(io)
       end
"89a337ea6f60a4fd58999ab73dea099e41032138"

julia> open(GzipDecompressorStream, artifact) do io
           mktempdir() do tmp
               Tar.extract(io, tmp)
               tree_hash = bytes2hex(Pkg.GitTools.tree_hash(tmp))
           end
       end
"ed75b82e0dd9b53c4ac4e70376f3e6f330c72767"

julia> versioninfo()
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 24 × 12th Gen Intel(R) Core(TM) i9-12900K
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, alderlake)
  Threads: 1 on 24 virtual cores

[nhz2]

Yes, I agree it is unfortunate that new Julia users experienced this issue. One of my favorite features of Julia is how easy it is to install complex packages. I use a fully updated Windows 11, and "Developer Mode" is off by default for me as well. I'm curious, what do you require adwaita_icon_theme for? Funnily enough, the last commit on the upstream repo is titled adwaita icon theme: you're better off not using it

As to how this ended up getting released, I think the main issue is that GitHub actions and other automated tests didn't catch this issue at all. Many package developers do not have a Windows computer, they rely on GitHub actions to test that their packages can be installed and function on Windows for Julia prereleases. For example, look at https://github.com/m3g/Packmol.jl/commits/main/ all tests for "Julia nightly - windows-latest - x64" are green. From what I can tell this is because GitHub actions are not using a default version of Windows that most users have. GitHub set it up so that symlinks and maybe other things are allowed. I have tried to find a setting in GitHub actions to allow running automated tests on a typical user version of Windows, but I have yet to find anything. Do you have any ideas on how to improve automated package testing on Windows?

[revolutionflourlake772]

Do you have any ideas on how to improve automated package testing on Windows?

As I alluded to somewhat, advanced automated testing frameworks aren't exactly my wheelhouse :sweat_smile: . Certainly, I make some nifty programs that are wildly beyond my peer group, but my peer group doesn't write software at all lol.

I understand this is maybe not "ideal", but given the gap here, I guess my hardware engineer mindset is "just actually test doing the thing if it is easy to do". In this case, someone with an updated Windows PC can just try to install a very bloated environment on a fresh .julia folder (which is what I did :sweat_smile: ) and see how it goes, including mass-"using" the packages after instantiating and updating. Also, I know the TTLX (time to load X?, whatever we want to call importing) development benchmarking has been done with some "meta-package" that has a ton of things in it. I guess I would say, just try to install and use that.

In my test, the fully passing 1.10 new release was broken within a couple minutes. I understand this is not necessarily a satisfying answer, and that there are limitations in its universality, but it is also a brute empirical approach that is extremely low effort, and which seemingly was not tried. I will add once more that this issue, here, #3643, existed 2.5 months before the release, so while it fell off the radar, it was not unknown.

I'm not saying that it wouldn't be better to automatically test out all this. However, as someone who has a pretty limited-but-growing skillset in the domain, I guess I'm just used to running into barriers that would take too much business time to make "pretty", when I am being paid to solve a problem (and I shoehorn programming into it as much as I can). So a checklist with a few extremely minimal cost manual steps is not a deal breaker for me (and imho is preferable to being almost a month past an official 1.X Julia release not working out-of-the-box on what is by far the most used PC/laptop OS overall, even if not by developers).

[StefanKarpinski]

I think the first thing to do here is partially revert the change where we started checking hashes on Windows—if we cannot create symlinks, then don't check installed tarball hashes. Yes, that's not great, but it's what we were doing before. That is quick and easy and fixes the situation on Windows for now. And can definitely go into 1.10.1.

The next thing to do is to add a check that the Tar.tree_hash value is correct on all platforms. That protects from getting the wrong tarball on all platforms and is sufficient for 1.10 going forward. If it gets into 1.10.1, great, if not then it can go into 1.10.x for some larger value of x.

For 1.11, we can add the Tar.tree_hash(; copy_symlinks) option and then use that to check the extracted hash even when we can't copy symlinks. This is extra paranoid, requires adding a new Tar API feature, and is not necessary for 1.10.

[StefanKarpinski]

The answer to "how was this released?" is that this only breaks on very specific artifacts on specific versions of Windows and even though https://github.com/JuliaLang/Pkg.jl/issues/3643 existed, it was not at all clear why that hash check was failing. I did look into it but wasn't able to figure it out. There are many reasons why one specific person's file system could be doing weird stuff, and it wasn't at all clear at that point that this issue was going to be widespread, nor what the specific circumstances for the issue were. That didn't become clear until after the release. If I'd had easy access to Windows 10 machines to debug on, I might have been able to make more progress, but I don't and no one else stepped up to look at the issue or debug on a Windows 10 system.

In order for this to have been caught in a reproducible, debuggable way by pre-release, we would have to test essentially all packages on essentially all supported versions of all supported OSes. Because I can't see any other way that we would happen to test the specific combination of installing these specific JLLs on Windows 10 or Windows 11. That's a staggering amount of compute resources that we don't really have.

[mkitti]

Perhaps we need a Julia on Windows Quality Assurance team? I would be happy to try to organize it. It would be a great if the team was composed of both volunteers and paid staff.

[sloede]

Perhaps we need a Julia on Windows Quality Assurance team? I would be happy to try to organize it. It would be a great if the team was composed of both volunteers and paid staff.

Sounds like a terrific idea! Maybe as a start it would be helpful to document (and, possibly provide) Windows cloud instances with which a developer can try to reproduce and eventually fix issues that only occur on Windows. For me, the only Windows platform currently available is a GHA Windows runner with the tmate-action, which is extremely annoying to use for any real debugging...

[mkitti]

I think the first thing to do here is partially revert the change where we started checking hashes on Windows—if we cannot create symlinks, then don't check installed tarball hashes. Yes, that's not great, but it's what we were doing before. That is quick and easy and fixes the situation on Windows for now. and can go into 1.10.1.

I'm not sure if full reversion is necessary. We could just demote the issue to a warning on Windows for 1.10.1. During testing this should error though so it is clearly seen in continuous integration. Basically we just need some time to fix the artifacts and build the infrastructure to detect symlinks.

Ultimately, no actual fix in Julia, Tar.jl, or Pkg.jl may be necessary. This is really an artifact packaging issue specific to Windows, so we just need to fix the artifacts, but this will take some time to address properly. Trying to emulate the file layout by copying symlinks is only a partial solution.

[StefanKarpinski]

I really don’t like unactionable warnings. What is the user supposed to do with that warning? I also don’t see that we necessarily want to disallow symlinks on Windows. After all, they do work on more recent windows versions. I agree that it’s good to discourage symlinks on Windows but the copying hack does work well enough most of the time that I think flat out disallowing is not the right approach.

Regarding a Windows QA team, that’s a great idea. There’s not any money for paid staff, so it’s going to have to be a volunteer operation. Re:

document (and, possibly provide) Windows cloud instances with which a developer can try

This is a lot of work tbh. I don’t know any core Julia devs who have the time, inclination and skills to do this. If someone who uses Windows knows how to do this and is willing, it would be a great contribution. Short of that, it would be helpful if people who use Windows would test Julia releases before they are final. That’s exactly what alphas, betas and release candidates are meant to be for. But it would seem that Windows users don’t actually test those or we would have more than a single report of this issue prior to the release. Anyway, maybe creating a Windows QA team would serve to encourage Windows users who have agreed to do QA to actually try out prereleases and report issues.

[mkitti]

What is the user supposed to do with that warning?

Either report the issue, reconfigure their system, or ignore it to own future peril. Perhaps it should be something to the effect of "Hey, this package is going to break in Julia 1.11. Either enable Developer Mode or file an issue with the package maintainer".

I also don’t see that we necessarily want to disallow symlinks on Windows. After all, they do work on more recent windows versions.

This requires that that Developer Mode be enabled as was discussed above. Microsoft does not want to enable this for everyone by default because malware can exploit symlinks to create security issues for naive software.

The following dialog box is quite scary for users, and I'm not sure if all Julia users would consider themselves professional developers.

The warning on Windows pull request has already been drafted.

I agree that it’s good to discourage symlinks on Windows but the copying hack does work well enough most of the time that I think flat out disallowing is not the right approach.

Nathan has convinced me that the copy symlinks issue is actually quite complicated and that we cannot support all possible symlink configurations as he demonstrated in https://github.com/JuliaIO/Tar.jl/pull/167#issuecomment-1890747309 . While we end up in a stack overflow, I'm going to guess that other software may fare much worse.

Calculating tree hashes in Tar files is probably important nonetheless, but it is a far from a complete solution on Windows. Until Microsoft comes up with a way for symlinks not to be an exploitation vector in Windows, we need to disallow deploying symlinks to Windows users.

document (and, possibly provide) Windows cloud instances with which a developer can try

This is a lot of work tbh. I don’t know any core Julia devs who have the time, inclination and skills to do this. If someone who uses Windows knows how to do this and is willing, it would be a great contribution. Short of that, it would be helpful if people who use Windows would test Julia releases before they are final. That’s exactly what alphas, betas and release candidates are meant to be for. But it would seem that Windows users don’t actually test those or we would have more than a single report of this issue prior to the release. Anyway, maybe creating a Windows QA team would serve to encourage Windows users who have agreed to do QA to actually try out prereleases and report issues.

I develop mainly on Windows because my clients and users use Windows. My primary development machine for work is not the best place to be testing out alpha or beta stage software that I did not write or do not have a directed interest in. That said I do have access to local and remote virtual machines for testing, but they also do not exist for this purpose.

Some cloud providers do provide a Virtual Windows Desktop. A Windows QA team could coordinate access to such a service.

There is definitely a gap between Windows users and Windows developers, especially ones who develop on Julia. Many Windows users do not have a Github account or otherwise know how to submit a proper bug report. If we do get a report from a pre-release I think we really need to follow up on it.

It seems to me that you have no one to delegate the investigation of the issue to. A Windows QA team would solve that issue. Moreover, I think a Windows QA team with resources would be able to fix such issues before release.

[lmiq]

After all, they do work on more recent windows versions.

FWIW, I was getting this error in a brand new Windows 11 laptop. I'm not sure this was clear.

[StefanKarpinski]

Ok, let's take the discussion of a Windows QA team elsewhere (Slack or Discourse) and keep this focused on the specific technical issue. Unless someone objects, this is the plan:

1. 1.10.1: Stop checking tree hashes at all on Windows if we can't create symlinks
2. 1.10.x: Add check that Tar.tree_hashvalue is correct on all platforms
3. 1.11.0: add Tar.tree_hash(; copy_symlinks) and use it to check expected hash

[nhz2]

Short of that, it would be helpful if people who use Windows would test Julia releases before they are final.

Finding this issue requires not just testing the prereleases of Julia, but also deleting the artifacts directory, otherwise, Julia will not try and install new artifacts. This issue is primarily affecting users who are using Julia, or maybe programming for the first time in 1.10 because they do not have existing artifacts installed.

The copy symlink hack is in principle fine, but this has to be documented as the expected behavior. Looking at https://pkgdocs.julialang.org/v1/artifacts/ there isn't anything about ignoring tree hashes on Windows. Because of this, I think the change to verifying tree hashes on Windows is a bug fix and will help improve the reliability of testing packages on Windows moving forward, even if it may break some people's workflows today.

I think your plan is good, but it should be implemented in such a way that when testing the next prerelease I don't have to delete my artifacts directory again, which might be tricky to pull off.

This is important because things like https://github.com/julia-actions/cache may assume the artifacts directory can be cached between Julia patch releases.

[mkitti]

FWIW, I was getting this error in a brand new Windows 11 laptop. I'm not sure this was clear.

Does enabling Developer Mode resolve the issue?

[PetrKryslUCSD]

Perhaps we need a Julia on Windows Quality Assurance team? I would be happy to try to organize it. It would be a great if the team was composed of both volunteers and paid staff.

I'd be happy to help. I use Win 11 (almost) exclusively (sometimes I execute Julia sims on serious hardware).

[PetrKryslUCSD]

FWIW, I was getting this error in a brand new Windows 11 laptop. I'm not sure this was clear.

Does enabling Developer Mode resolve the issue?

Apparently it does: I substituted Developer Mode for export JULIA_PKG_IGNORE_HASHES=true, and the failures did not come back (fresh install of the depot from scratch). But frankly, allowing Windows to potentially open the doors to malicious software does not fill me with a warm glow.

[StefanKarpinski]

Part 1 has dropped: https://github.com/JuliaLang/Pkg.jl/pull/3764. Can I get someone here with a Windows system to test this? You need to do the following:

1. Clone the Pkg repo
2. Check out https://github.com/JuliaLang/Pkg.jl/pull/3764
3. Edit the uuid entry in the Pkg/Project.toml file, changing any digit
4. Set the JULIA_DEPOT_PATH variable to an empty temp directory
5. Start Julia 1.10 in the Pkg directory with julia --project
6. Try doing ] add P4est_jll@v2.8.1

If step 4 gives problems, you might need to delete an old Manifest.toml file in the Pkg repo.

[sloede]

Part 1 has dropped: https://github.com/JuliaLang/Pkg.jl/pull/3764.

Great, thanks a lot! Ideally, anyone testing this should probably first verify that installing P4est_jll as-is actually does fail on their system...

@joshualampert @lchristm @arseniykholod Does any one of you have a Windows system with which you can reproduce the original issue, i.e., triggering the artifact hash mismatch error on Julia v1.10? If you do, it would be great if you can try out Stefan's fix and report the results here 🙏

[JoshuaLampert]

I don't have any Windows system. Sorry.

[lmiq]

Does enabling Developer Mode resolve the issue?

Yes, it does (Windows 11).

6. Try doing ] add P4est_jll@v2.8.1

I think I followed the instructions, but here I just get:

(Pkg.jl) pkg> add P4est_jll@v2.8.1
ERROR: Could not parse project UUID as a UUID

I changed a digit of the UUID there, as instructed.

[DilumAluthge]

I changed a digit of the UUID there, as instructed.

When changing Pkg.jl's UUID, try editing it exactly as follows. If this is the original TOML file:

name = "Pkg"
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

Try changing it to exactly this:

name = "Pkg"
uuid = "54cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[lmiq]

Then I get the same previous error:

(Pkg) pkg> add P4est_jll@v2.8.1
   Resolving package versions...
  Downloaded artifact: OpenSSL
  Downloaded artifact: Hwloc
  Downloaded artifact: MicrosoftMPI
  Downloaded artifact: libevent
  Downloaded artifact: P4est
  Downloaded artifact: P4est
ERROR: Unable to automatically download/install artifact 'P4est' from sources listed in 'C:\Users\leand\.julia\packages\P4est_jll\kfjxs\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767
- https://github.com/JuliaBinaryWrappers/P4est_jll.jl/releases/download/P4est-v2.8.1+2/P4est.v2.8.1.x86_64-w64-mingw32-mpi+microsoftmpi.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767

Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\Artifacts.jl:422
  [3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\Operations.jl:755
  [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\Operations.jl:1394
  [5] add
    @ Pkg.Operations C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\Operations.jl:1376 [inlined]
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::@Kwargs{io::Base.TTY})
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:278
  [7] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{})
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:159
  [8] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:148
  [9] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\REPLMode\REPLMode.jl:412
 [10] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\REPLMode\REPLMode.jl:390
 [11] do_cmd
    @ Pkg.REPLMode C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\REPLMode\REPLMode.jl:380 [inlined]
 [12] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\REPLMode\REPLMode.jl:557
 [13] #invokelatest#2
    @ Base .\essentials.jl:887 [inlined]
 [14] invokelatest
    @ Base .\essentials.jl:884 [inlined]
 [15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\REPL\src\LineEdit.jl:2656
 [16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\REPL\src\REPL.jl:1312
 [17] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\REPL\src\REPL.jl:386

(Pkg) pkg>

(I think that the package does not install with the original UUID either, if I delete the Artifacts folder)

[lmiq]

Just to be sure, this means I´m in the correct branch, right?

C:\Users\leand\Downloads\Pkg.jl>git status
HEAD detached at 83b34499d

[IanButterworth]

Did you import Pkg before switching to pkg mode? You should see Pkg itself precompile

[lmiq]

Now I did, and it didn´t precompile. I think I got into some messy state:

C:\Users\leand\Downloads\Pkg.jl>del Manifest.toml
Não foi possível encontrar C:\Users\leand\Downloads\Pkg.jl\Manifest.toml

C:\Users\leand\Downloads\Pkg.jl>julia +1.10 --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0 (2023-12-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import Pkg
    Updating registry at `C:\Users\leand\.julia\registries\General.toml`
    Updating `C:\Users\leand\Downloads\Pkg.jl\Project.toml`
  [f43a241f] ~ Downloads ⇒ v1.6.0
  [ea8e919c] ~ SHA ⇒ v0.7.0
  [fa267f1f] ~ TOML ⇒ v1.0.3
  [a4e569a6] ~ Tar ⇒ v1.10.0
  [3f19e933] ~ p7zip_jll ⇒ v17.4.0+2
    Updating `C:\Users\leand\Downloads\Pkg.jl\Manifest.toml`
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling Pkg
        Info Given Pkg was explicitly requested, output will be shown live
ERROR: LoadError: UndefVarError: `generating_output` not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base .\Base.jl:31
 [2] top-level scope
   @ C:\Users\leand\Downloads\Pkg.jl\src\precompile.jl:167
 [3] include(mod::Module, _path::String)
   @ Base .\Base.jl:495
 [4] include(x::String)
   @ Pkg C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:3
 [5] top-level scope
   @ C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:841
 [6] include
   @ Base .\Base.jl:495 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base .\loading.jl:2216
 [8] top-level scope
   @ stdin:3
in expression starting at C:\Users\leand\Downloads\Pkg.jl\src\precompile.jl:6
in expression starting at C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:3
in expression starting at stdin:3
  ✗ Pkg
  0 dependencies successfully precompiled in 4 seconds

ERROR: The following 1 direct dependency failed to precompile:

Pkg [54cfe95a-1eb2-52ea-b672-e2afdf69b78f]

Failed to precompile Pkg [54cfe95a-1eb2-52ea-b672-e2afdf69b78f] to "C:\\Users\\leand\\.julia\\compiled\\v1.10\\Pkg\\jl_BED5.tmp".
ERROR: LoadError: UndefVarError: `generating_output` not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base .\Base.jl:31
 [2] top-level scope
   @ C:\Users\leand\Downloads\Pkg.jl\src\precompile.jl:167
 [3] include(mod::Module, _path::String)
   @ Base .\Base.jl:495
 [4] include(x::String)
   @ Pkg C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:3
 [5] top-level scope
   @ C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:841
 [6] include
   @ Base .\Base.jl:495 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base .\loading.jl:2216
 [8] top-level scope
   @ stdin:3
in expression starting at C:\Users\leand\Downloads\Pkg.jl\src\precompile.jl:6
in expression starting at C:\Users\leand\Downloads\Pkg.jl\src\Pkg.jl:3
in expression starting at stdin:
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\Types.jl:70
  [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{…}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, _from_loading::Bool, kwargs::@Kwargs{…})
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:1656
  [3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{_from_loading::Bool})
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:159
  [4] precompile
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:147 [inlined]
  [5] #precompile#114
    @ Pkg.API C:\Users\leand\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Pkg\src\API.jl:146 [inlined]
  [6] #invokelatest#2
    @ Base .\essentials.jl:889 [inlined]
  [7] invokelatest
    @ Base .\essentials.jl:884 [inlined]
  [8] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:1957
  [9] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1806
 [10] #invoke_in_world#3
    @ Base .\essentials.jl:921 [inlined]
 [11] invoke_in_world
    @ Base .\essentials.jl:918 [inlined]
 [12] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1797
 [13] macro expansion
    @ Base .\loading.jl:1784 [inlined]
 [14] macro expansion
    @ Base .\lock.jl:267 [inlined]
 [15] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1747
 [16] #invoke_in_world#3
    @ Base .\essentials.jl:921 [inlined]
 [17] invoke_in_world
    @ Base .\essentials.jl:918 [inlined]
 [18] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1740
Some type information was truncated. Use `show(err)` to see complete types.

julia>

[IanButterworth]

Ah.. we need to make a backport branch for this to work on 1.10

Can you try the same but on julia nightly

[lmiq]

It worked as expected, I think:

C:\Users\leand\Downloads\Pkg.jl>"C:\Users\leand\AppData\Local\Programs\Julia-1.11.0-DEV\bin\julia.exe" --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.1289 (2024-01-18)
 _/ |\__'_|_|_|\__'_|  |  Commit 8562af5552 (0 days old master)
|__/                   |

julia> import Pkg
┌ Info: Precompiling Pkg [54cfe95a-1eb2-52ea-b672-e2afdf69b78f]
└ @ Base loading.jl:2568

(Pkg) pkg> add P4est_jll@v2.8.1
   Resolving package versions...
┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
│   Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767
│
│ $JULIA_PKG_IGNORE_HASHES is set to 1: ignoring error and moving artifact to the expected location
└ @ Pkg.Artifacts C:\Users\leand\Downloads\Pkg.jl\src\Artifacts.jl:340
      Compat entries added for P4est_jll
    Updating `C:\Users\leand\Downloads\Pkg.jl\Project.toml`
  [6b5a15aa] + P4est_jll v2.8.1+2
  [56f22d72] ~ Artifacts ⇒ v1.11.0
  [ade2ca70] ~ Dates ⇒ v1.11.0
  [f43a241f] ~ Downloads ⇒ v1.6.0
  [7b1f6079] ~ FileWatching ⇒ v1.11.0
  [76f85450] ~ LibGit2 ⇒ v1.11.0
  [8f399da3] ~ Libdl ⇒ v1.11.0
  [56ddb016] ~ Logging ⇒ v1.11.0
  [d6f4376e] ~ Markdown ⇒ v1.11.0
  [de0858da] ~ Printf ⇒ v1.11.0
  [3fa0cd96] ~ REPL ⇒ v1.11.0
  [9a3f8284] ~ Random ⇒ v1.11.0
  [ea8e919c] ~ SHA ⇒ v0.7.0
  [9e88b42a] ~ Serialization ⇒ v1.11.0
  [fa267f1f] ~ TOML ⇒ v1.0.3
  [a4e569a6] ~ Tar ⇒ v1.10.0
  [cf7118a7] ~ UUIDs ⇒ v1.11.0
  [3f19e933] ~ p7zip_jll ⇒ v17.4.0+2
    Updating `C:\Users\leand\Downloads\Pkg.jl\Manifest.toml`
  [692b3bcd] + JLLWrappers v1.5.0
  [3da0fdf6] + MPIPreferences v0.1.10
  [21216c6a] + Preferences v1.4.1
  [e33a78d0] + Hwloc_jll v2.10.0+0
  [7cb0a576] + MPICH_jll v4.1.2+1
  [f1f71cc9] + MPItrampoline_jll v5.3.1+1
  [9237b28f] + MicrosoftMPI_jll v10.1.4+1
  [fe0851c0] + OpenMPI_jll v5.0.1+0
  [458c3c95] + OpenSSL_jll v3.0.12+0
  [6b5a15aa] + P4est_jll v2.8.1+2
  [32165bc3] + PMIx_jll v4.2.7+0
  [1080aeaf] + libevent_jll v2.1.13+1
  [eb928a42] + prrte_jll v3.0.2+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching v1.11.0
  [b77e0a4c] + InteractiveUtils v1.11.0
  [4af54fe1] + LazyArtifacts v1.11.0
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] + Libdl v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.11.0
  [de0858da] + Printf v1.11.0
  [3fa0cd96] + REPL v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization v1.11.0
  [6462fe0b] + Sockets v1.11.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] + LibCURL_jll v8.5.0+0
  [e37daf67] + LibGit2_jll v1.7.1+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.6+0
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.58.0+0
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  14 dependencies successfully precompiled in 15 seconds. 28 already precompiled.

(Pkg) pkg> st
Project Pkg v1.11.0
Status `C:\Users\leand\Downloads\Pkg.jl\Project.toml`
  [6b5a15aa] P4est_jll v2.8.1+2
  [56f22d72] Artifacts v1.11.0
  [ade2ca70] Dates v1.11.0
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching v1.11.0
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [cf7118a7] UUIDs v1.11.0
  [3f19e933] p7zip_jll v17.4.0+2

(Pkg) pkg>

[IanButterworth]

Do you have that env var set? If so unset it

[lmiq]

No I don´t. The only environment variable related to Julia is the one asked here:

(I´m not a Windows user either, so if there some other place to check, please let me know)

[StefanKarpinski]

The way the PR works is that it treats the variable defaulting to true as equivalent to the variable being set. Maybe that's too confusing; if so I can rework it.

[StefanKarpinski]

Yeah, I'll change it so the error messages are different.

[ArseniyKholod]

Part 1 has dropped: #3764.

Great, thanks a lot! Ideally, anyone testing this should probably first verify that installing P4est_jll as-is actually does fail on their system...

I'm not sure, that I understood correctly, but adding the P4est_jll doesn't fail by me using Julia 1.10.0. The only difference with original issue I see, that I'm using Julia 1.10 and not beta version, and I use Windows 11, not 10th.

(@v1.10) pkg> add P4est_jll
    Updating registry at C:\Users\Lenovo\.julia\registries\General.toml
   Resolving package versions...
    Updating C:\Users\Lenovo\.julia\environments\v1.10\Project.toml
  [6b5a15aa] + P4est_jll v2.8.1+2
    Updating C:\Users\Lenovo\.julia\environments\v1.10\Manifest.toml
  [3da0fdf6] + MPIPreferences v0.1.10
  [e33a78d0] + Hwloc_jll v2.10.0+0
  [7cb0a576] + MPICH_jll v4.1.2+1
  [f1f71cc9] + MPItrampoline_jll v5.3.1+1
  [9237b28f] + MicrosoftMPI_jll v10.1.4+1
  [fe0851c0] + OpenMPI_jll v5.0.1+0
  [6b5a15aa] + P4est_jll v2.8.1+2
  [32165bc3] + PMIx_jll v4.2.7+0
  [1080aeaf] + libevent_jll v2.1.13+1
  [eb928a42] + prrte_jll v3.0.2+0
  [4af54fe1] + LazyArtifacts
Precompiling project...
  10 dependencies successfully precompiled in 11 seconds. 151 already precompiled.

julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39 (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 6 × AMD Ryzen 5 4500U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
  Threads: 1 on 6 virtual cores

Did I understand correctly what I suppose to do?

[mkitti]

@ArseniyKholod are you running as administrator or have you enabled Windows developer mode?

[IanButterworth]

3764 is now queued up for 1.10.1 https://github.com/JuliaLang/julia/pull/52755

[StefanKarpinski]

Thanks to @IanButterworth we now have Part 1 ready to go into 1.10.1 which means the time-critical part is done. 🥳 🎉 🎈

The next step that would be nice to do soonish would be to add unconditional checking of Tar.tree_hash for all tarballs that we download. I'm not sure if that warrants changing anything that was done in part 1; after all, it's still something of an issue that some tarballs get created with copied files instead of symlinks, so warning might be good. Then again, there's no attack vector since we know the tarballs is right, it just can't be extracted faithfully. I think the best option is probably to leave the warning so that people complain about artifacts with symlinks that end up on Windows systems.

[ArseniyKholod]

@ArseniyKholod are you running as administrator or have you enabled Windows developer mode?

No, as long as I understand, Developer Mode is disabled in settings, Windows Powershell is running in user mode, not like administrator (in Task Manager Powershell has NO in Elevated column).

[nhz2]

3764 introduces a new bug on Windows where installing an artifact can delete an existing artifact.

Steps to reproduce: In one julia environment run ]add iso_codes_jll@4.15.1.

Then in a second environment run

(jl_tANRC2) pkg> add iso_codes_jll@4.15.0
   Resolving package versions...
┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   695a3abd0af051dab62687501b9a61ad91c831a3
│   Calculated git-tree-sha1: 1719102ef324df43b9fe123a231c2ee630931a03
│ 
│ System is Windows and user cannot create symlinks: ignoring hash mismatch and moving artifact to the expected location   
└ @ Pkg.Artifacts C:\Users\nzimm\github\nhz2\Pkg.jl\src\Artifacts.jl:346

Finally back in the first environment:

julia> using iso_codes_jll
ERROR: InitError: Artifact "iso_codes" was not found by looking in the paths:
  C:\Users\nzimm\.julia\artifacts\1719102ef324df43b9fe123a231c2ee630931a03
  C:\Users\nzimm\Downloads\julia-latest-win64\julia-ec686c3cb7\local\share\julia\artifacts\1719102ef324df43b9fe123a231c2ee630931a03
  C:\Users\nzimm\Downloads\julia-latest-win64\julia-ec686c3cb7\share\julia\artifacts\1719102ef324df43b9fe123a231c2ee630931a03
Try `using Pkg; Pkg.instantiate()` to re-install all missing resources if the artifact is part of a package or call `Pkg.ensure_artifact_installed` (https://pkgdocs.julialang.org/v1/api/#Pkg.Artifacts.ensure_artifact_installed) if not.

This happens because if the hash is ignored.

https://github.com/JuliaLang/Pkg.jl/blob/b4318759029a91cee42e7e5bca91e8bc9a0d453d/src/Artifacts.jl#L348

will delete src which can be a valid artifact.

[IanButterworth]

This issue should be fixed on master and the upcoming v1.10.1 but if anyone wants to double check https://discourse.julialang.org/t/julia-1-10-1-testing-period/109805

[lmiq]

Is there a windows binary for the release candidate available anywhere?

[Devansh1106]

The same error was reportedly encountered by @jmark on the Trixi.jl Slack.

I also the same in Trixi.jl . Where can I get the solution for this? I am using Julia v1.10

[mkitti]

Which point release of Julia? 1.10.2 shod have addressed this.

[Devansh1106]

I am using v1.10.2 . The issue occurred when I was setting up Trixi.jl in my local machine. I ran this command. julia> using Pkg julia> Pkg.add(["Trixi", "Trixi2Vtk", "OrdinaryDiffEq", "Plots"]) This is what I got:

[IanButterworth]

That issue appears to be an unrelated issue with permissions of your known_hosts file.

[IanButterworth]

I think this issue can be closed. There are some next steps proposed here https://github.com/JuliaLang/Pkg.jl/issues/3643#issuecomment-1904782538 but the current situation seems reasonable of a warning that will work but be noisy enough to promote people not using symlinks in windows artifacts.

[Devansh1106]

That issue appears to be an unrelated issue with permissions of your known_hosts file.

Do you have any idea for how can I rectify the permission issue? I think the error I am getting maybe related to the warning.

[IanButterworth]

Please open another issue for that

[StefanKarpinski]

Actually, please don't open an issue, post a question on Julia discourse.

[Devansh1106]

I have posted in Julia discourse.

[giordano]

@jgreener64 has just run into the "Tree Hash Mismatch!" error with Julia v1.10.2 in NVTX_jll@v3.1.0+2, I thought this was solved in Julia v1.10.2?

Edit: actually, I don't see symlinks in https://github.com/JuliaBinaryWrappers/NVTX_jll.jl/releases/download/NVTX-v3.1.0%2B2/NVTX.v3.1.0.x86_64-w64-mingw32.tar.gz, so the problem may be different :thinking:

[StefanKarpinski]

@giordano: can you open a new issue for that one with info about the tree hash and how to reproduce?

[jgreener64]

I opened a new issue at https://github.com/JuliaLang/Pkg.jl/issues/3856.