search

ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.23k stars 350 forks source link

Windows: Symlinks in opam packages will sometimes fail #5406

Open jonahbeckford opened 1 year ago

jonahbeckford commented 1 year ago

The essential problem is that dangling symlinks are not supported in MSYS2 (and sometimes not in Cygwin), and symlinks in general aren't portable in native Windows. Simply based on the order of tar members in url { src: ".../xyz.tar.gz" } a symlink can become a dangling symlink.

This is a tricky problem. One instance is https://github.com/thierry-martinez/stdcompat/issues/27 which has full details.

One solution was the PR I did in https://github.com/ocaml/opam/pull/4813. That did two passes for tar ... the first would do everything except the symlinks and the second pass would do all the files. But somehow or for some reason that didn't make it into the code base. https://github.com/ocaml/opam/compare/2b60006117b116612138929e5e1059189e86f96f..7d40698858f27b9d9b01e0183c882a76d04398ce#diff-e9267480037ccaf8a2614178ed1aa887d1ae3e6cd5d986a62a420f5a61344fa3L613-L652 was that solution.

My PR above is specific to MSYS2 and slow. An alternative solution is to fix the tar archives during opam publish and dune-release.

dra27 commented 1 year ago

The third option (at least for opam itself) is ocaml-tar: i.e. we avoid using an external program completely. There's various other reasons for wanting to do that for performance - most especially, it would allow us to process index.tar.gz without having to blat tens of thousands of small files on NTFS...!

Gbury commented 1 year ago

Note: I've also seem to hit this bug in my CI runs on github actions with ocamlbuild.0.14.2 (see this log). Weirdly enough this appeared out of nowhere recently, even though neither tar not ocamlbuild have had recent releases/changes.