search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
621 stars 261 forks source link

Artifact hash mismatch with P4est_jll on Julia v1.10 #3643

Closed stillyslalom closed 6 months ago

stillyslalom commented 1 year ago

I'm encountering an error when trying to install P4est_jll in Julia v1.10-beta2 (installed via juliaup) on two different Windows 10 machines - the issue is not encountered on v1.9.3:

(jl_tdZRQD) pkg> add P4est_jll
   Resolving package versions...
  Downloaded artifact: P4est
  Downloaded artifact: P4est
ERROR: Unable to automatically download/install artifact 'P4est' from sources listed in 'C:\Users\Alex\.julia\packages\P4est_jll\kfjxs\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767
- https://github.com/JuliaBinaryWrappers/P4est_jll.jl/releases/download/P4est-v2.8.1+2/P4est.v2.8.1.x86_64-w64-mingw32-mpi+microsoftmpi.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767

Troubleshooting on Slack with @sloede and @fredrikekre:

julia> using Tar, CodecZlib, Downloads, SHA

julia> artifact = Downloads.download("https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138")
"C:\\Users\\Alex\\AppData\\Local\\Temp\\jl_glqlGnAL6b"

julia> shasum = bytes2hex(open(SHA.sha256, artifact))
"1abe2cd4e2c31d34c840d3042f94d6b34d85762c6be0fbd480eaf52136b566e1"

julia> open(GzipDecompressorStream, artifact) do io
           tree_hash = Tar.tree_hash(io)
       end
"89a337ea6f60a4fd58999ab73dea099e41032138"

julia> open(GzipDecompressorStream, artifact) do io
           mktempdir() do tmp
               Tar.extract(io, tmp)
               tree_hash = bytes2hex(Pkg.GitTools.tree_hash(tmp))
           end
       end
"ed75b82e0dd9b53c4ac4e70376f3e6f330c72767"
julia> versioninfo()
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 24 × 12th Gen Intel(R) Core(TM) i9-12900K
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, alderlake)
  Threads: 1 on 24 virtual cores
StefanKarpinski commented 12 months ago

I want to ask you to do this:

open(GzipDecompressorStream, artifact) do io
    mktempdir() do tmp
        Tar.extract(io, tmp)
        run(`ls -Rl $tmp`)
    end
end

However, I don't know if you have GNU ls available to run. I've asked ChatGPT what the equivalent on Windows is and it's told me that it's this in PowerShell:

Get-ChildItem -Recurse | ForEach-Object { 
    $acl = Get-Acl $_.FullName 
    [PSCustomObject]@{
        Path = $_.FullName
        Access = $acl.AccessToString
    }
} | Format-List

which, is cool, but man that's a lot of text to replicate ls -lR. If you have a better way to see the names and permissions of files in the directory, that's what I'd like to see. Maybe something with walkdir...

stillyslalom commented 12 months ago

Blessedly, I have ls available.

julia> open(GzipDecompressorStream, artifact) do io
           mktempdir() do tmp
               Tar.extract(io, tmp)
               run(`ls -Rl $tmp`)
           end
       end
'C:\Users\355547\AppData\Local\Temp\jl_UxWajc':
total 48
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 bin
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 etc
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 include
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 lib
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 share

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/bin':
total 20580
-rwxr-xr-x 1 355547 1049089 5000947 Oct  5 10:45 libp4est-0.dll
-rwxr-xr-x 1 355547 1049089 1484704 Oct  5 10:45 libsc-0.dll
-rwxr-xr-x 1 355547 1049089  348593 Oct  5 10:45 p4est_balance_seeds.exe
-rwxr-xr-x 1 355547 1049089  349393 Oct  5 10:45 p4est_bricks.exe
-rwxr-xr-x 1 355547 1049089  348695 Oct  5 10:45 p4est_loadconn.exe
-rwxr-xr-x 1 355547 1049089  367400 Oct  5 10:45 p4est_mesh.exe
-rwxr-xr-x 1 355547 1049089  525780 Oct  5 10:45 p4est_particles.exe
-rwxr-xr-x 1 355547 1049089  354389 Oct  5 10:45 p4est_points.exe
-rwxr-xr-x 1 355547 1049089  342534 Oct  5 10:45 p4est_read_conn.exe
-rwxr-xr-x 1 355547 1049089  363179 Oct  5 10:45 p4est_simple.exe
-rwxr-xr-x 1 355547 1049089  425164 Oct  5 10:45 p4est_spheres.exe
-rwxr-xr-x 1 355547 1049089  352976 Oct  5 10:45 p4est_step1.exe
-rwxr-xr-x 1 355547 1049089  344928 Oct  5 10:45 p4est_step2.exe
-rwxr-xr-x 1 355547 1049089  398509 Oct  5 10:45 p4est_step3.exe
-rwxr-xr-x 1 355547 1049089  410119 Oct  5 10:45 p4est_step4.exe
-rwxr-xr-x 1 355547 1049089  377076 Oct  5 10:45 p4est_timings.exe
-rwxr-xr-x 1 355547 1049089  346793 Oct  5 10:45 p4est_write_conn.exe
-rwxr-xr-x 1 355547 1049089  349613 Oct  5 10:45 p8est_balance_seeds.exe
-rwxr-xr-x 1 355547 1049089  350417 Oct  5 10:45 p8est_bricks.exe
-rwxr-xr-x 1 355547 1049089  349719 Oct  5 10:45 p8est_loadconn.exe
-rwxr-xr-x 1 355547 1049089  367924 Oct  5 10:45 p8est_mesh.exe
-rwxr-xr-x 1 355547 1049089  526289 Oct  5 10:45 p8est_particles.exe
-rwxr-xr-x 1 355547 1049089  338854 Oct  5 10:45 p8est_periodicity.exe
-rwxr-xr-x 1 355547 1049089  355421 Oct  5 10:45 p8est_points.exe
-rwxr-xr-x 1 355547 1049089  343558 Oct  5 10:45 p8est_read_conn.exe
-rwxr-xr-x 1 355547 1049089  348076 Oct  5 10:45 p8est_read_tetgen.exe
-rwxr-xr-x 1 355547 1049089  358196 Oct  5 10:45 p8est_simple.exe
-rwxr-xr-x 1 355547 1049089  431071 Oct  5 10:45 p8est_spheres.exe
-rwxr-xr-x 1 355547 1049089  353484 Oct  5 10:45 p8est_step1.exe
-rwxr-xr-x 1 355547 1049089  345440 Oct  5 10:45 p8est_step2.exe
-rwxr-xr-x 1 355547 1049089  397485 Oct  5 10:45 p8est_step3.exe
-rwxr-xr-x 1 355547 1049089  411915 Oct  5 10:45 p8est_step4.exe
-rwxr-xr-x 1 355547 1049089  378810 Oct  5 10:45 p8est_timings.exe
-rwxr-xr-x 1 355547 1049089  439973 Oct  5 10:45 p8est_tsearch.exe
-rwxr-xr-x 1 355547 1049089  347305 Oct  5 10:45 p8est_write_conn.exe
-rwxr-xr-x 1 355547 1049089  333415 Oct  5 10:45 sc_b64dec.exe
-rwxr-xr-x 1 355547 1049089  334105 Oct  5 10:45 sc_b64enc.exe
-rwxr-xr-x 1 355547 1049089  345619 Oct  5 10:45 sc_function.exe
-rwxr-xr-x 1 355547 1049089  339716 Oct  5 10:45 sc_logging.exe
-rwxr-xr-x 1 355547 1049089  347320 Oct  5 10:45 sc_options.exe
-rwxr-xr-x 1 355547 1049089  356141 Oct  5 10:45 sc_test_shmem.exe

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/etc':
total 8
-rw-r--r-- 1 355547 1049089 845 Oct  5 10:45 Makefile.p4est.mk
-rw-r--r-- 1 355547 1049089 689 Oct  5 10:45 Makefile.sc.mk

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/include':
total 1280
-rw-r--r-- 1 355547 1049089 23307 Oct  5 10:45 p4est.h
-rw-r--r-- 1 355547 1049089 16731 Oct  5 10:45 p4est_algorithms.h
-rw-r--r-- 1 355547 1049089  2966 Oct  5 10:45 p4est_balance.h
-rw-r--r-- 1 355547 1049089 20031 Oct  5 10:45 p4est_base.h
-rw-r--r-- 1 355547 1049089 34567 Oct  5 10:45 p4est_bits.h
-rw-r--r-- 1 355547 1049089  4641 Oct  5 10:45 p4est_build.h
-rw-r--r-- 1 355547 1049089 30523 Oct  5 10:45 p4est_communication.h
-rw-r--r-- 1 355547 1049089  8860 Oct  5 10:45 p4est_config.h
-rw-r--r-- 1 355547 1049089 38830 Oct  5 10:45 p4est_connectivity.h
-rw-r--r-- 1 355547 1049089  2499 Oct  5 10:45 p4est_empty.h
-rw-r--r-- 1 355547 1049089 34898 Oct  5 10:45 p4est_extended.h
-rw-r--r-- 1 355547 1049089  4509 Oct  5 10:45 p4est_geometry.h
-rw-r--r-- 1 355547 1049089 19546 Oct  5 10:45 p4est_ghost.h
-rw-r--r-- 1 355547 1049089  3508 Oct  5 10:45 p4est_io.h
-rw-r--r-- 1 355547 1049089 12167 Oct  5 10:45 p4est_iterate.h
-rw-r--r-- 1 355547 1049089 15629 Oct  5 10:45 p4est_lnodes.h
-rw-r--r-- 1 355547 1049089 16553 Oct  5 10:45 p4est_mesh.h
-rw-r--r-- 1 355547 1049089  5737 Oct  5 10:45 p4est_nodes.h
-rw-r--r-- 1 355547 1049089  4177 Oct  5 10:45 p4est_plex.h
-rw-r--r-- 1 355547 1049089  3561 Oct  5 10:45 p4est_points.h
-rw-r--r-- 1 355547 1049089 26698 Oct  5 10:45 p4est_search.h
-rw-r--r-- 1 355547 1049089  3174 Oct  5 10:45 p4est_spheres.h
-rw-r--r-- 1 355547 1049089 29012 Oct  5 10:45 p4est_to_p8est.h
-rw-r--r-- 1 355547 1049089  1291 Oct  5 10:45 p4est_to_p8est_empty.h
-rw-r--r-- 1 355547 1049089  1621 Oct  5 10:45 p4est_to_p8est_spheres.h
-rw-r--r-- 1 355547 1049089 17264 Oct  5 10:45 p4est_vtk.h
-rw-r--r-- 1 355547 1049089 18227 Oct  5 10:45 p4est_wrap.h
-rw-r--r-- 1 355547 1049089 29274 Oct  5 10:45 p6est.h
-rw-r--r-- 1 355547 1049089  4407 Oct  5 10:45 p6est_communication.h
-rw-r--r-- 1 355547 1049089  1297 Oct  5 10:45 p6est_empty.h
-rw-r--r-- 1 355547 1049089 15984 Oct  5 10:45 p6est_extended.h
-rw-r--r-- 1 355547 1049089  9520 Oct  5 10:45 p6est_ghost.h
-rw-r--r-- 1 355547 1049089 10719 Oct  5 10:45 p6est_lnodes.h
-rw-r--r-- 1 355547 1049089  4175 Oct  5 10:45 p6est_profile.h
-rw-r--r-- 1 355547 1049089  7595 Oct  5 10:45 p6est_vtk.h
-rw-r--r-- 1 355547 1049089 23489 Oct  5 10:45 p8est.h
-rw-r--r-- 1 355547 1049089 16778 Oct  5 10:45 p8est_algorithms.h
-rw-r--r-- 1 355547 1049089  3394 Oct  5 10:45 p8est_balance.h
-rw-r--r-- 1 355547 1049089 40686 Oct  5 10:45 p8est_bits.h
-rw-r--r-- 1 355547 1049089  4641 Oct  5 10:45 p8est_build.h
-rw-r--r-- 1 355547 1049089 29749 Oct  5 10:45 p8est_communication.h
-rw-r--r-- 1 355547 1049089 47887 Oct  5 10:45 p8est_connectivity.h
-rw-r--r-- 1 355547 1049089  1297 Oct  5 10:45 p8est_empty.h
-rw-r--r-- 1 355547 1049089 35147 Oct  5 10:45 p8est_extended.h
-rw-r--r-- 1 355547 1049089  5571 Oct  5 10:45 p8est_geometry.h
-rw-r--r-- 1 355547 1049089 19619 Oct  5 10:45 p8est_ghost.h
-rw-r--r-- 1 355547 1049089  3512 Oct  5 10:45 p8est_io.h
-rw-r--r-- 1 355547 1049089 16860 Oct  5 10:45 p8est_iterate.h
-rw-r--r-- 1 355547 1049089 18891 Oct  5 10:45 p8est_lnodes.h
-rw-r--r-- 1 355547 1049089 19357 Oct  5 10:45 p8est_mesh.h
-rw-r--r-- 1 355547 1049089  6731 Oct  5 10:45 p8est_nodes.h
-rw-r--r-- 1 355547 1049089  4177 Oct  5 10:45 p8est_plex.h
-rw-r--r-- 1 355547 1049089  3561 Oct  5 10:45 p8est_points.h
-rw-r--r-- 1 355547 1049089 26743 Oct  5 10:45 p8est_search.h
-rw-r--r-- 1 355547 1049089  3185 Oct  5 10:45 p8est_spheres.h
-rw-r--r-- 1 355547 1049089  3769 Oct  5 10:45 p8est_tets_hexes.h
-rw-r--r-- 1 355547 1049089 17263 Oct  5 10:45 p8est_vtk.h
-rw-r--r-- 1 355547 1049089 17253 Oct  5 10:45 p8est_wrap.h
-rw-r--r-- 1 355547 1049089 32738 Oct  5 10:45 sc.h
-rw-r--r-- 1 355547 1049089  8122 Oct  5 10:45 sc3_mpi_types.h
-rw-r--r-- 1 355547 1049089  2114 Oct  5 10:45 sc_allgather.h
-rw-r--r-- 1 355547 1049089  6175 Oct  5 10:45 sc_amr.h
-rw-r--r-- 1 355547 1049089  7362 Oct  5 10:45 sc_avl.h
drwxr-xr-x 1 355547 1049089     0 Oct  5 10:45 sc_builtin
-rw-r--r-- 1 355547 1049089 11071 Oct  5 10:45 sc_config.h
-rw-r--r-- 1 355547 1049089 48282 Oct  5 10:45 sc_containers.h
-rw-r--r-- 1 355547 1049089  5085 Oct  5 10:45 sc_flops.h
-rw-r--r-- 1 355547 1049089  4504 Oct  5 10:45 sc_functions.h
-rw-r--r-- 1 355547 1049089  1215 Oct  5 10:45 sc_getopt.h
-rw-r--r-- 1 355547 1049089 13983 Oct  5 10:45 sc_io.h
-rw-r--r-- 1 355547 1049089 10382 Oct  5 10:45 sc_keyvalue.h
-rw-r--r-- 1 355547 1049089 19929 Oct  5 10:45 sc_mpi.h
-rw-r--r-- 1 355547 1049089 17838 Oct  5 10:45 sc_notify.h
-rw-r--r-- 1 355547 1049089 18418 Oct  5 10:45 sc_options.h
-rw-r--r-- 1 355547 1049089  8133 Oct  5 10:45 sc_polynom.h
-rw-r--r-- 1 355547 1049089  1787 Oct  5 10:45 sc_private.h
-rw-r--r-- 1 355547 1049089  2917 Oct  5 10:45 sc_random.h
-rw-r--r-- 1 355547 1049089  5727 Oct  5 10:45 sc_ranges.h
-rw-r--r-- 1 355547 1049089  2685 Oct  5 10:45 sc_reduce.h
-rw-r--r-- 1 355547 1049089  5021 Oct  5 10:45 sc_refcount.h
-rw-r--r-- 1 355547 1049089  3262 Oct  5 10:45 sc_search.h
-rw-r--r-- 1 355547 1049089  6341 Oct  5 10:45 sc_shmem.h
-rw-r--r-- 1 355547 1049089  2436 Oct  5 10:45 sc_sort.h
-rw-r--r-- 1 355547 1049089 13504 Oct  5 10:45 sc_statistics.h
-rw-r--r-- 1 355547 1049089  4834 Oct  5 10:45 sc_string.h
-rw-r--r-- 1 355547 1049089 10903 Oct  5 10:45 sc_uint128.h
-rw-r--r-- 1 355547 1049089  3193 Oct  5 10:45 sc_unique_counter.h
-rw-r--r-- 1 355547 1049089  4979 Oct  5 10:45 sc_v4l2.h

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/include/sc_builtin':
total 16
-rw-r--r-- 1 355547 1049089 6125 Oct  5 10:45 getopt.h
-rw-r--r-- 1 355547 1049089 4766 Oct  5 10:45 getopt_int.h

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/lib':
total 800
-rw-r--r-- 1 355547 1049089 532658 Oct  5 10:45 libp4est.dll.a
-rw-r--r-- 1 355547 1049089 279814 Oct  5 10:45 libsc.dll.a
drwxr-xr-x 1 355547 1049089      0 Oct  5 10:45 pkgconfig

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/lib/pkgconfig':
total 2
-rw-r--r-- 1 355547 1049089 414 Oct  5 10:45 libsc-2.8.3.pc
-rw-r--r-- 1 355547 1049089 447 Oct  5 10:45 p4est-2.8.pc

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share':
total 12
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 aclocal
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 data
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 ini
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 licenses

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share/aclocal':
total 96
-rw-r--r-- 1 355547 1049089  8195 Oct  5 10:45 ax_prefix_config_h.m4
-rw-r--r-- 1 355547 1049089  1671 Oct  5 10:45 ax_split_version.m4
-rw-r--r-- 1 355547 1049089  2752 Oct  5 10:45 p4est_include.m4
-rw-r--r-- 1 355547 1049089  1406 Oct  5 10:45 p4est_metis.m4
-rw-r--r-- 1 355547 1049089  5561 Oct  5 10:45 p4est_petsc.m4
-rw-r--r-- 1 355547 1049089  1413 Oct  5 10:45 sc_builtin.m4
-rw-r--r-- 1 355547 1049089 11537 Oct  5 10:45 sc_include.m4
-rw-r--r-- 1 355547 1049089  3340 Oct  5 10:45 sc_memalign.m4
-rw-r--r-- 1 355547 1049089 16651 Oct  5 10:45 sc_mpi.m4
-rw-r--r-- 1 355547 1049089  1890 Oct  5 10:45 sc_openmp.m4
-rw-r--r-- 1 355547 1049089  5318 Oct  5 10:45 sc_package.m4
-rw-r--r-- 1 355547 1049089  1735 Oct  5 10:45 sc_pthread.m4
-rw-r--r-- 1 355547 1049089  2275 Oct  5 10:45 sc_qsort.m4
-rw-r--r-- 1 355547 1049089  1946 Oct  5 10:45 sc_v4l2.m4

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share/data':
total 394
-rw-r--r-- 1 355547 1049089  12762 Oct  5 10:45 hole_2d_cubit.inp
-rwxr-xr-x 1 355547 1049089    333 Oct  5 10:45 hole_2d_cubit.jou
-rw-r--r-- 1 355547 1049089    541 Oct  5 10:45 hole_2d_gmsh.geo
-rw-r--r-- 1 355547 1049089   8143 Oct  5 10:45 hole_2d_gmsh.inp
-rw-r--r-- 1 355547 1049089 179294 Oct  5 10:45 hole_3d_cubit.inp
-rwxr-xr-x 1 355547 1049089    335 Oct  5 10:45 hole_3d_cubit.jou
-rw-r--r-- 1 355547 1049089    626 Oct  5 10:45 hole_3d_gmsh.geo
-rw-r--r-- 1 355547 1049089 127609 Oct  5 10:45 hole_3d_gmsh.inp
-rw-r--r-- 1 355547 1049089  42424 Oct  5 10:45 p8est_box_tetgen.ele
-rw-r--r-- 1 355547 1049089   8560 Oct  5 10:45 p8est_box_tetgen.node

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share/ini':
total 6
-rw-r--r-- 1 355547 1049089 1235 Oct  5 10:45 sc_iniparser_twisted.ini
-rw-r--r-- 1 355547 1049089  239 Oct  5 10:45 sc_options_example.ini
-rw-r--r-- 1 355547 1049089   80 Oct  5 10:45 sc_options_preload.ini

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share/licenses':
total 0
drwxr-xr-x 1 355547 1049089 0 Oct  5 10:45 P4est

'C:\Users\355547\AppData\Local\Temp\jl_UxWajc/share/licenses/P4est':
total 20
-rw-r--r-- 1 355547 1049089 18092 Oct  5 10:45 COPYING
Process(`ls -Rl 'C:\Users\355547\AppData\Local\Temp\jl_UxWajc'`, ProcessExited(0))
stillyslalom commented 12 months ago

The same error was reportedly encountered by @jmark on the Trixi.jl Slack.

sloede commented 11 months ago

I'm getting similar tree hash mismatch errors when running PackageCompiler.jl CI tests on Windows, e.g. here

revolutionflourlake772 commented 9 months ago

As @giordano linked above, I'm encountering a similar error trying to use 1.10 for the first time today. In addition to the adwaita_icon_theme_jll error, I'm also getting one on iso_codes_jll upon re-trying to instantiate an hour later:

(JL1XVENV) pkg> up
    Updating registry at `C:\1\.julia\registries\General.toml`
  Downloaded artifact: iso_codes
  Downloaded artifact: iso_codes
ERROR: Unable to automatically download/install artifact 'iso_codes' from sources listed in 'C:\1\.julia\packages\iso_codes_jll\wcUPz\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/71f68a3d55d73f2e15a3969c241fae2349b1feb5
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   71f68a3d55d73f2e15a3969c241fae2349b1feb5
  Calculated git-tree-sha1: 409d6ac4c02dae43ff4fe576b5c5820d0386fb3f
- https://github.com/JuliaBinaryWrappers/iso_codes_jll.jl/releases/download/iso_codes-v4.11.0+0/iso_codes.v4.11.0.any.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   71f68a3d55d73f2e15a3969c241fae2349b1feb5
  Calculated git-tree-sha1: 409d6ac4c02dae43ff4fe576b5c5820d0386fb3f
lmiq commented 9 months ago

I would also like to add that this is preventing the installation of the NativeFileDialog.jl and Gtk.jl packages in Julia 1.10. It is preventing the installation of a package of mine (Packmol.jl), which uses that such to help newbie users. Path to reproduce:

  1. Remove the .julia/artifacts folder
  2. Install (on Julia 1.10) the `NativeFileDialog" package.

Currently the workaround I'm suggesting to users is to use Julia 1.9.

lmiq commented 9 months ago

I'd also note that iso_codes_jll is an artifact with ~1.6k monthly downloads, so this issue might be causing problems for a lot of 1.10 users on Windows. If I can provide any info that helps solving the issue, please let me know.

IanButterworth commented 9 months ago

As a workaround in the interim you should be able to set env var JULIA_PKG_IGNORE_HASHES=1 but I'd use it sparingly.

lmiq commented 9 months ago

Yes, I know. But for the kind of user I'm suggesting the pop-up of a file dialog that's unfortunately not an option.

nhz2 commented 9 months ago

I was able to work around this issue by turning on "Developer Mode" in the "For developers" settings on Windows 11. The issue with iso_codes_jll is that it contains symlinks. On Windows from what I can tell creating symlinks requires "Developer Mode" to be on.

giordano commented 9 months ago

But why is that not a problem until Julia v1.9?

IanButterworth commented 9 months ago

Seems we need to bisect this?

nhz2 commented 9 months ago

The main change affecting this from v1.9 is https://github.com/JuliaLang/Pkg.jl/pull/3349

IanButterworth commented 9 months ago

As in you've tested 1.10.0 with and without that commit? fyi @staticfloat

nhz2 commented 9 months ago

No, I haven't tested 1.10.0 without that commit.

lmiq commented 9 months ago

I've tested with 1.10.0-alpha1 and the error is there (and not on 1.9.4).

IanButterworth commented 9 months ago

A helpful thing to do would be to open a PR that adds a failing test for this

IanButterworth commented 9 months ago

Has anyone tried a reproducer with JULIA_DEBUG=Pkg on? There's a debug log in the artifact install step that might shed some light.

nhz2 commented 9 months ago

I'm able to reproduce the bad hash for iso_codes on Linux using ENV["BINARYPROVIDER_COPYDEREF"]="true":

julia> ENV["BINARYPROVIDER_COPYDEREF"]="true"
"true"

julia> using Pkg

julia> Pkg.add("iso_codes_jll")
   Resolving package versions...
  Downloaded artifact: iso_codes
  Downloaded artifact: iso_codes
ERROR: Unable to automatically download/install artifact 'iso_codes' from sources listed in '/home/nathan/.julia/packages/iso_codes_jll/wcUPz/Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/71f68a3d55d73f2e15a3969c241fae2349b1feb5
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   71f68a3d55d73f2e15a3969c241fae2349b1feb5
  Calculated git-tree-sha1: 409d6ac4c02dae43ff4fe576b5c5820d0386fb3f
- https://github.com/JuliaBinaryWrappers/iso_codes_jll.jl/releases/download/iso_codes-v4.11.0+0/iso_codes.v4.11.0.any.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   71f68a3d55d73f2e15a3969c241fae2349b1feb5
  Calculated git-tree-sha1: 409d6ac4c02dae43ff4fe576b5c5820d0386fb3f
mkitti commented 9 months ago

Reverting 0fb323e57b4569e105a1bb47bf1975d3c60b4197 seems to have resolved the problem.

~/src/Pkg.jl $ ../../julia/usr/bin/julia
(@v1.11) pkg> activate --temp
  Activating new project at `D:\msys64\tmp\jl_riX32k`

(jl_riX32k) pkg> add P4est_jll
   Resolving package versions...
ERROR: Unable to automatically download/install artifact 'P4est' from sources listed in 'C:\Users\kittisopikulm\.julia
\packages\P4est_jll\kfjxs\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767
- https://github.com/JuliaBinaryWrappers/P4est_jll.jl/releases/download/P4est-v2.8.1+2/P4est.v2.8.1.x86_64-w64-mingw32
-mpi+microsoftmpi.tar.gz
    Error: Tree Hash Mismatch!
  Expected git-tree-sha1:   89a337ea6f60a4fd58999ab73dea099e41032138
  Calculated git-tree-sha1: ed75b82e0dd9b53c4ac4e70376f3e6f330c72767

~/src/Pkg.jl $ git revert 0fb323e57b4569e105a1bb47bf1975d3c60b4197

~/src/Pkg.jl $ ../../julia/usr/bin/julia --project=.
(Pkg) pkg> activate --temp
  Activating new project at `D:\msys64\tmp\jl_lIrCJs`

(jl_lIrCJs) pkg> add P4est_jll
   Resolving package versions...
    Updating `D:\msys64\tmp\jl_lIrCJs\Project.toml`
  [6b5a15aa] + P4est_jll v2.8.1+2
IanButterworth commented 9 months ago

Ok. Can you try without the revert, with JULIA_DEBUG=Pkg

IanButterworth commented 9 months ago

So before https://github.com/JuliaLang/Pkg.jl/commit/0fb323e57b4569e105a1bb47bf1975d3c60b4197 which landed in 1.10.0 we didn't check the treehash of artifacts on Windows.

I think there are two ways forward:

  1. Revert https://github.com/JuliaLang/Pkg.jl/commit/0fb323e57b4569e105a1bb47bf1975d3c60b4197 and go back to not checking treehashes of artifacts on Windows
  2. Go with guiding the user to fix the symlink issue https://github.com/JuliaLang/Pkg.jl/issues/3745 which mentions JULIA_PKG_IGNORE_HASHES if symlinks aren't working, in case the user cannot turn developer mode on.

@staticfloat

staticfloat commented 9 months ago

I'd like to continue checking treehashes on windows. Supposedly, most Windows 10 installations (after the "creators update") should support symlinks without Administrator privileges, and our libuv should be making use of that feature.

I like the approach taken in https://github.com/JuliaLang/Pkg.jl/pull/3745 as it steers people toward the right way to fix things, without undoing some of our integrity/security work.

Another possibility is to change the iso_codes_jll build process to create a windows-specific tarball that has no symlinks.

giordano commented 9 months ago

Another possibility is to change the iso_codes_jll build process to create a windows-specific tarball that has no symlinks.

That doesn't really address the issue though: there may be plenty of other artifacts not created by BinaryBuilder which contain symlinks.

sloede commented 9 months ago

Is there a way to check how many packages are affected by this issue, ie, if it would be feasible to contact all actively maintained packages' developers?

In general, IMHO one of the unique selling propositions of Julia is its excellent, low-effort portability to all major platforms (from a user perspective). Thus, if there is an "ugly" but functional solution (assuming that not checking the hash still results in working packages), I'd rather have users be able to use JLLs that were created "wrongly", instead of being strict about it and scaring less experienced users away. Or at least allow for a grace period in which package maintainers have a chance of fixing their JLLs

mkitti commented 9 months ago

My recommendation is that on Windows, perhaps below a certain version, that make the tree hash mismatch just a warning if we detect that symlinks are an issue. Julia did not check tree hash mismatches before, and we need to provide some time for users and developers to adapt.

sloede commented 9 months ago

My recommendation is that on Windows, perhaps below a certain version, that make the tree hash mismatch just a warning if we detect that symlinks are an issue. Julia did not check tree hash mismatches before, and we need to provide some time for users and developers to adapt.

A warning sounds reasonable to me!

staticfloat commented 9 months ago

My perspective is that hash verification is not a nice-to-have, it's a fundamental property of our storage system, and violating it should only be done if we can guarantee that there are no ill effects. Bypassing hash checking should be treated the same as Base.@assume_effects; occasionally it may be necessary for a good user experience, but you should always think twice before doing it. That being said, once you've figured out that it is needed, it should be employed if no better option is available.

In this case, it may seem that just always dereferencing symlinks and generating files is the correct thing to do, but it can have undesirable effects (e.g. busybox and git both do things like symlink many different commands to the same binary, so dereferencing the symlinks would explode disk usage to many times the original amount) all the way to completely incorrect behavior (e.g. if you used to have one library and symlinks that point to that library, you can start to have multiple copies of a library loaded if they are actually copies).

We have so far avoided these issues by ensuring that windows JLLs just don't have symlinks. iso_codes_jll is an exception because it is packaged as an "any" platform JLL, and so doesn't hit the windows-specific symlink manual copyderef code in BinaryBuilder.

I propose that we keep the check being strict, add @IanButterworth's message in https://github.com/JuliaLang/Pkg.jl/pull/3745, and then add a windows-specific tarball to iso_codes_jll that doesn't contain symlinks. Our experience so far is that there are not very many artifacts on windows that contain symlinks thanks to our BB pre-processing, and windows users are increasingly going to be on versions of Windows that will support these symlinks automatically; if you believe this Windows 10 and 11 make up ~95% of the windows world.

I'm not saying that we shouldn't support older windows versions, but that I think the best use of effort is to make the popular packages/artifacts "just work" by avoiding symlinks, and make it easy for older systems to work around the issue.

My recommendation is that on Windows, perhaps below a certain version, that make the tree hash mismatch just a warning if we detect that symlinks are an issue.

I'd be fine with this, but it might be a little difficult; the condition hash_mismatch && cannot_create_symlinks is insufficient, you need to know that the reason the hash check failed is because you could not create a symlink, otherwise you open a hole where any hash failure gets installed on older windows systems. (And while yes, this was the previous behavior, it's undesirable IMO).

lmiq commented 9 months ago

add a windows-specific tarball to iso_codes_jll

In any case is this easy? Can it be done as a first step to make the packages that we know are failing work again?

nhz2 commented 9 months ago

I've gone through the _jll packages with this script and found the following have artifacts with symlinks that are installed on Windows:

Edit: I've also gone through the non jll packages.

I'm downloading for only one platform in the script, so I may have missed some symlinks.

mkitti commented 9 months ago

My perspective is that hash verification is not a nice-to-have, it's a fundamental property of our storage system, and violating it should only be done if we can guarantee that there are no ill effects.

Overall, I agree. The question is how quickly can we get the Julia 1.10 ecosystem to a state where it is not broken on Windows. Is it just a matter of going through the list above or is there a larger issue?

giordano commented 9 months ago

Is it just a matter of going through the list above or is there a larger issue?

My understanding is that we can't have artifacts with symlinks on Windows and reliably check their integrity on disk. I want to reiterate jlls are only a part of the artifacts, but not an exclusive component, fixing them isn't going to fix the underlying issue.

mkitti commented 9 months ago

I'd be fine with this, but it might be a little difficult; the condition hash_mismatch && cannot_create_symlinks is insufficient, you need to know that the reason the hash check failed is because you could not create a symlink

Could we include an alternate hash for when the archives are expanded without symlinks?

julia> using Tar, CodecZlib, Downloads, SHA

julia> function tree_hash_without_symlinks(url)
           artifact = Downloads.download(url)
           open(GzipDecompressorStream, artifact) do io
               mktempdir() do tmp
                   Tar.extract(io, tmp)
                   # Remove all symlinks
                   for (root, dirs, files) in walkdir(tmp)
                       for file in files
                           file_path = joinpath(root, file)
                           islink(file_path) && rm(file_path)
                       end
                   end
                   tree_hash = bytes2hex(Pkg.GitTools.tree_hash(tmp))
               end
           end
       end
tree_hash_without_symlinks (generic function with 1 method)

julia> tree_hash_without_symlinks("https://pkg.julialang.org/artifact/89a337ea6f60a4fd58999ab73dea099e41032138")
"ed75b82e0dd9b53c4ac4e70376f3e6f330c72767"

Then if the calculated tree hash matches the alternate hash, then we know what the problem is: the symlinks are missing.

mkitti commented 9 months ago

In https://github.com/JuliaIO/Tar.jl/pull/165, I implemented a keyword to skip symlinks within the tar file. This allows us to calculate the hash as if the symlinks did not exist like the example above. Unlike the example above, this does not depend on extracting the tar file itself.

julia> using CodecZlib, Tar

julia> open(GzipDecompressorStream, "P4est.v2.8.1.x86_64-w64-mingw32-mpi+microsoftmpi.tar.gz") do io
           tree_hash = Tar.tree_hash(io)
       end
"89a337ea6f60a4fd58999ab73dea099e41032138"

julia> open(GzipDecompressorStream, "P4est.v2.8.1.x86_64-w64-mingw32-mpi+microsoftmpi.tar.gz") do io
           tree_hash = Tar.tree_hash(io; skip_symlink=true)
       end
"ed75b82e0dd9b53c4ac4e70376f3e6f330c72767"

My suggestion is implement the following procedure.

  1. Detect the hash mismatch
  2. Report the mismatch
  3. Calculate the tarfile hash without symlinks as in my Tar.jl pull request
  4. If the calculated hash matches the tarfile hash without symlinks, issue a warning if on a platform where this is a known issue.
  5. Otherwise throw an error.
staticfloat commented 9 months ago

My understanding is that we can't have artifacts with symlinks on Windows and reliably check their integrity on disk.

That is true on machines where users are unable to create symlinks, but as I pointed out above, that should be <5% of all users right now, and continuing to decrease, as more and more users upgrade to Windows 10 or 11.

In https://github.com/JuliaIO/Tar.jl/pull/165, I implemented a keyword to skip symlinks within the tar file

I think it might actually be a little simpler than this; if we just scan the tarball and see that there's a symlink in there, and we can't create symlinks, that might be a sufficient condition for skipping the hash check entirely. So if you instead implement Tar.has_symlink(tarball) or something like that, that could be enough?

mkitti commented 9 months ago

That is true on machines where users are unable to create symlinks, but as I pointed out above, that should be <5% of all users right now, and continuing to decrease, as more and more users upgrade to Windows 10 or 11.

Not all Windows 10 users have access to symlinks. My Windows 10 machine certainly does not.

I think it might actually be a little simpler than this; if we just scan the tarball and see that there's a symlink in there, and we can't create symlinks, that might be a sufficient condition for skipping the hash check entirely. So if you instead implement Tar.has_symlink(tarball) or something like that, that could be enough?

I think Tar.has_symlink is certainly possible given what I just implemented. However, I do not actually think this is sufficient to indicate that the symlink issue is the only issue present. For example, would we be able to distinguish the lack of symlinks from some kind of faulty disk or file system that is corrupting bytes? Or perhaps someone is attempting to create a portable Julia install on a NTFS or FAT32 formatted USB flash drive via Linux?

mkitti commented 9 months ago

I'm coming around to Tar.has_symlink because I realized that the symlink issue may involve copying rather than simply skipping symlinks.

sloede commented 8 months ago

That is true on machines where users are unable to create symlinks, but as I pointed out above, that should be <5% of all users right now, and continuing to decrease, as more and more users upgrade to Windows 10 or 11.

Not all Windows 10 users have access to symlinks. My Windows 10 machine certainly does not.

IIRC, the error appears (or can appear) also on windows-latest on GHA, but I might misremember

cc @joshualampert

mkitti commented 8 months ago

Note that in the case for P4est_jll, we probably want to find an alternate approach: https://github.com/JuliaIO/Tar.jl/pull/165#issuecomment-1880295075

This line is fishy: https://github.com/JuliaPackaging/Yggdrasil/blob/4d82df7a64f9535fdba24119de35cb4c1eecc999/P/P4est/build_tarballs.jl#L51

mkitti commented 8 months ago

I implemented Tar.has_symlink here: https://github.com/JuliaIO/Tar.jl/pull/166

StefanKarpinski commented 8 months ago

I'd like to advocate for switching to checking the tree hash of the tarball before extraction instead of after we've extracted. We have that capability and it's much more reliable. For security, it's just as good since it allows us to verify that we've got the right content before trying to extract it. It doesn't verify that we've done the extraction and installation correctly. One way to look at this issue is that this error showing up actually proves that checking the on-disk tree hash is an effective way to check if we've installed something correctly—since we've copied symlinks instead of creating them and technically that's wrong. I would propose this logic:

  1. Download the tarball to a temporary location, tarball_path.
  2. Check the tree hash of the downloaded tarball using Tar.tree_hash(tarball_path).
  3. Determine expected_hash based on the following logic: 
    expected_hash = tarball_hash
if Sys.iswindows() && Tar.contains(:symlink) && !can_create_symlinks(artifacts_dir)
 expected_hash = Tar.tree_hash(tarball_path, copy_symlinks=true)
end
  4. Extract the tarball in a temp location under artifacts_dir
  5. Check that the extracted tarball has the expected tree hash.
  6. Move the installed tree from the temp location to the final location atomically.
  7. If we want to be really paranoid we could check the on-disk tree hash one last time.

The alternative that @staticfloat has advocated for is to just skip the check of the on-disk tree hash in the case where the expected hash will be different from the tarball tree hash.

StefanKarpinski commented 8 months ago

The other thing we can do is just check Tar.tree_hash and skip checking the extracted on-disk tree hash.

fredrikekre commented 8 months ago

checking the tree hash of the tarball before extraction

I was going to suggest this too (at least on Windows then). We already have the utility for it since we do this for registry tarballs: https://github.com/JuliaLang/Pkg.jl/pull/3408

mkitti commented 8 months ago

Don't we already check the hash of the entire artifact (the tarball)? Does checking the tree hash of the same tarball really help that much?

staticfloat commented 8 months ago

Don't we already check the hash of the entire artifact (the tarball)? Does checking the tree hash of the same tarball really help that much?

On the one hand, we check the sha256 of the tarball itself when we download it from the origin server, but if we download it from a caching layer (such as a pkgserver) we do not because the tarball will have been repacked. The repacking will change the tarball itself but not that contents.

nhz2 commented 8 months ago

I've replaced the symlinks with copies of the files that they point to in iso_codes in https://github.com/JuliaPackaging/Yggdrasil/pull/7926

StefanKarpinski commented 8 months ago

Even in cases where we can check both hashes, it's good to have multiple layers of defense. One could have the correct sha256 hash of a download tarball and yet accidentally have a tarball with the wrong git tree hash. Checking both ensures that such mistakes aren't allowed.

I would like to point out here that whatever fix we initially get out for this needs to be done entirely in Pkg and in the tarballs that are uploaded since we shouldn't make a patch release of Tar in Julia 1.10 that adds features—semver applies to stdlibs also.

mkitti commented 8 months ago

PATCH version when you make backward compatible bug fixes

I would argue that the solution would be a "backward compatible bug fix" even if it ended up adding a new feature to fix the bug.

nhz2 commented 8 months ago

There is an issue with just logging a warning and then moving the artifact to the final location if there is an artifact hash mismatch due to symlink copying. If the user sees the initial warning what can they do to fix the issue? If they turn "Developer Mode" on and reinstall the package, the incorrect artifact will already be installed so the issue won't be fixed. I assume they would also have to manually delete their "~/.julia/artifacts" directory.

Also, if they ignore the initial warning, when they restart Julia, how would they know anything is wrong?

A potential solution could be to instead install an artifact with a wrong hash in a different place, like "~/.julia/bad_artifacts", and then show a warning in the output of pkg> status if an artifact is being used from "~/.julia/bad_artifacts".

I think https://github.com/JuliaLang/Pkg.jl/pull/3745 is a better way forward. When using Windows, I don't expect every package or artifact to install properly. If I get an error I can switch to using WSL. In my opinion, having a subtly broken artifact is worse, because I won't know to switch to using WSL until I start having weird unexpected behavior much later on.

revolutionflourlake772 commented 8 months ago

My perspective is that hash verification is not a nice-to-have, it's a fundamental property of our storage system, and violating it should only be done if we can guarantee that there are no ill effects.

Overall, I agree. The question is how quickly can we get the Julia 1.10 ecosystem to a state where it is not broken on Windows. Is it just a matter of going through the list above or is there a larger issue?

So, not to get too off track here, but having rebooted this issue on the first day of 1.10 Windows availability, I feel it is worth briefly going back to what I wrote in the Discourse thread that I made about it (https://discourse.julialang.org/t/cannot-instantiate-env-in-julia-1-10-adwaita-icon-theme-jll-git-tree-hash-error-anyone-else/108105/2):

"It seems like somewhat of a “show-stopper” type of issue for a new version (unless I am somehow very unique), so certainly I wouldn’t want this to be experienced by newer users (I :heart: Julia and want it to gain popularity!)."

Given that we have acknowledged that 1.10 is indeed broken, at least out of the box, in Windows 10+, I wonder: how did this release? There was indeed this issue from a few months ago, and it seems that it didn't receive much attention. Perhaps I am in a minority of Windows users, but at a high level I feel like "major 1.X update is widely broken across the package ecosystem for all Windows users" should indeed be a show-stopper.

I'm quite an amateur / hardware engineer and don't mean this as a dig in any way. I really want Julia to succeed! I understand this is stating the obvious and I don't mean to belabor the point, but the "https://julialang.org/downloads/" current stable version being broken for Windows users seems like quite an obstacle to success for the language. Again as an amateur, this had been my first experience with the language a couple years ago, fair or not it would have been a damaging impression.

Also, above it was mentioned that Windows 10 and 11 should automatically support symlinks: is that a future-state default? I have fully updated Windows 10, and it seems that numerous other individuals who have Win 10/11 have had the same issues as me. I see "developer mode" is listed as a solution; that's fine for my PC here, but on my work laptop the setting is locked down. A practice like "JULIA_PKG_IGNORE_HASHES=1" seems quite undesirable, so I've just stayed on 1.9.4. To be clear, 1.9.4. is great, so it's not a big issue for me; my points are more referencing a generic Windows user, perhaps new to Julia or relatively new.