ability to override packages locally

[andrewrk]

20178 is not technically a prerequisite for this, but the two proposals would work well together.

It would be nice if you could do something like this:

$ cd my-zig-project
$ git clone git://example.com/zlib/ .zig-cache/p/$(zig pkg-hash zlib)

or

$ cd my-zig-project
$ ln -s $HOME/path-to-local-zlib-checkout/ .zig-cache/p/$(zig pkg-hash zlib)

then you could make edits to .zig-cache/p/<pkg hash> that would override the global packages, as well as make it easy to interact with the upstream source control of the dependency.

Crucially, these directories are typically not tracked by source control, making the workflow convenient to test a patch to a dependency.

When building with local overrides, a warning should be issued:

$ zig build
warning: overriding 'zlib' dependency with './zig-cache/p/zlib-1.3.1-3-IQwAAPXlgi9M'

When building in --system mode, it ignores local overrides with a warning:

$ zig build --system /usr/lib/zig/p
warning: ignoring local override './zig-cache/p/zlib-1.3.1-3-IQwAAPXlgi9M' in system mode

zig pkg-hash --list would list all the immediate dependency packages and their hashes:

$ zig pkg-hash --list
libz    zlib-1.3.1-3-IQwAAPXlgi9M
libvorbis    libvorbis-1.3.8-3-NQ8kAD5eWxrE

Then you could drill down into the sub-dependencies:

$ zig pkg-hash --list libvorbis
libvorbis.libz    zlib-1.3.1-3-IQwAAPXlgi9M

Or maybe --list would just print the whole tree.

Perhaps another subcommand could do the path calculation, so even if you're in a subdirectory, it would give you the local override path:

andy@ark:~/my-project/build$ zig lop libvorbis.libz
../.zig-cache/p/zlib-1.3.1-3-IQwAAPXlgi9M
andy@ark:~/my-project/build$ zig lop libvorbis.libztypo
error: no dependency package named 'libvorbis.libztypo'
info: use `zig pkg-hash --list` to browse dependency tree

This would be a way to quickly find out the command line to pass to e.g. git clone.

Another part of this workflow would be finding out the list of overrides that have mismatching hashes. Perhaps:

• zig lop --clean - delete all local overrides with mismatching hashes
• zig lop --clear - delete all local overrides
• zig lop --status - list all local overrides, their expected hashes, and actual hashes (when mismatching)

Related:

• 14288, which introduces the concept of a "project ID".

[scottredig]

This last week I've been working on a small game using https://github.com/scottredig/zig-javascript-bridge/ where this would have been helpful, and have some thoughts which may be relevant here:

Firstly, problems started after I did a release of zjb and found that the release was broken. I have examples in the zjb project which build correctly, however it turns out that I improperly exposed an artifact from build.zig. So slightly tangential to this issue, but what is exposed from std.Build.dependency is hard to test from within. (maybe the example should have its own build.zig, despite being in the same git repro?) Doing a release only to realize it was completely broken wasn't great.

My solution was to temporarily change the game's build.zig.zon to reference my local copy of zjb. With this proposal, using a local override would be done instead (so the game commit history doesn't assume my local project organization).

From there when I was using zjb to make a game in practice, I have found several features needed to be added (as I expected would happen). My cycle has been to add the feature to zjb, test it with the game, and when ready commit both.

However this has been small feedback loop. Therefor what I haven't done is gone back to zjb and made a release for every change I've made. Instead I'm waiting to find all of the changes I'm going to find to be in, and then do a single release.

The issue I've found with my approach, and would be an issue with this proposal as well, is this:

When working in a tight iteration loop, where one package is having direct influence on changes to another package, a tradeoff needs to be made: Either the dependency needs to have a semantic version release after every change made to it, or the project which imports the dependency will have a range of commits where history is broken.

I think in a more ideal world, the zig package manager would let me directly reference the desired commit in the dependency. Then the process would be:

1. Add local override.
2. Make changes in both the dependency and the project.
3. Commit dependency.
4. Update project's zon to reference commit made in 3, and commit project.
5. Repeat steps 2-4 until no more changes are needed.
6. Semantic version release of dependency.
7. Update project to use semantic version and commit.

It also wouldn't surprised me if I'm missing something about versioning with the package manager. I am quite new to using it.

[BratishkaErik]

Do I understand it correctly:

• Packages from local cache dir override packages from global cache dir,
• packages from system dir override both of them?

zig pkg-hash --list

Perhaps it would make sense to add additional --source or --origin parameter that will add third "source" column, which will show where package is taken from, I think it will be helpful to troubleshoot these patches. Something like:

libz      zlib-1.3.1-3-IQwAAPXlgi9M         local override
libvorbis libvorbis-1.3.8-3-NQ8kAD5eWxrE    global cache (or system dir if in system mode)

Inspired by command git config --list --show-scope:

system rebase.autosquash=true
system credential.helper=helper-selector
global core.editor=emacs
local core.symlinks=false
local core.ignorecase=true 

(example is from https://batsov.com/articles/2021/11/14/display-git-configuration/)

[andrewrk]

I think in a more ideal world, the zig package manager would let me directly reference the desired commit in the dependency. Then the process would be:

Note that this is possible via something like this:

            .url = "https://github.com/scottredig/zig-javascript/bridge/archive/502361981484f80ed11cbb66eee5be496a4aefa8.tar.gz",

For non-GitHub URLs the equivalent git+https:// scheme works and can reference individual commits.

[andrewrk]

@BratishkaErik Yes you understand correctly. --system <dir> provides <dir> as the one and only location where packages are used. Do I understand correctly that this is generally the desired behavior from Linux distro maintainers?

[scottredig]

Tangential to my post above: I would prefer that instead of .zig-cache/p/$(zig pkg-hash zlib) containing the dependency, .zig-cache/p/$(zig pkg-hash zlib).txt contain a path to a local copy of the dependency. (txt extension irrelevant to proposal) The file contents being only the file path with no other formatting would be the simplest.

This would have a few advantages:

1. No need to perform a local clone if you already have the project.
2. Any setup the user has for working with the repo (eg, editor having project local settings) can be used instead needing to do setup again.
3. Works better when you're primarily doing work on the dependency, but want to quickly test a project which uses the dependency /before/ committing.
4. Multiple projects using the dependency can be testing in one go.

[andrewrk]

A text file containing only a path is extremely similar to a symlink. Why reinvent symlinks? All 4 of your points work the same with actual symlinks.

[BratishkaErik]

@BratishkaErik Yes you understand correctly. --system <dir> provides <dir> as the one and only location where packages are used. Do I understand correctly that this is generally the desired behavior from Linux distro maintainers?

Yes for 99% of cases. If, for some reasons, package:

1. has ".zig-cache" dir with override added to source,
2. and the patches are package-dependent,
3. and the patch is too intrusive to be applied to dependency supplied by distro before building this package,

maintainers can always:

1. Copy system dir to another location and replace dependency with patched version here,
2. or introduce another package like libfoobar-downstream_name-patched and use it as a dependency in their package manager.

I think situation where all three conditions are met will be extremely rare. "1-st + 2-nd condition only" should be more often, but in this case it's likely this patch should be upstreamed, for benefits of each party (distros, upstream and downstream)

[andrewrk]

has ".zig-cache" dir with override added to source,

Note that this will never be the case. The .zig-cache directory must be excluded from source control, and must not be included in a package.

[BratishkaErik]

has ".zig-cache" dir with override added to source,

Note that this will never be the case. The .zig-cache directory must be excluded from source control, and must not be included in a package.

Must as in "Zig must return error if this will be the case" or must as in "Authors must refrain from this, but no hard checks"? Because if this is the second, chances are very small but still not zero, and downstream still has opportunity to commit specifically ".zig-cache/p/dep-1.2.3-sizedhash" folder and ".gitignore" everything else from ".zig-cache". Maybe most distros will have policy to reject these packages, but I can't find out in this proposal if Zig will also check ".gitignore".

[scottredig]

A text file containing only a path is extremely similar to a symlink. Why reinvent symlinks? All 4 of your points work the same with actual symlinks.

Fair point. I was mostly thinking "I want to do .path = "../zig-javascript-bridge/", in build.zig.zon, but just locally and with a warning."

A file containing the path would be consistent on different OSs, which would be an advantage for any tooling.

There is some friction on Windows worth considering:

• I've never made a symlink on Windows. My first thought was "does windows even have symlinks?" Some searching says yes, but it's not exactly a tool I'm used to working with, as opposed to just referencing a filepath.
• The default shell to open, PowerShell, doesn't understand mklink.
• It requires admin or switching the computer into dev mode (which requires admin). I don't think anything else I've done with Zig has required admin. This might be insurmountable for students, users of public computers, or developers in corporations with unaccommodating IT departments.

Which is all pretty stupid (on par for Windows). All that said, if the proposal goes through as is, it's nothing I wouldn't just deal with.

[andrewrk]

• It requires admin or switching the computer into dev mode (which requires admin). I don't think anything else I've done with Zig has required admin.

Yeah, this is something I've been extremely careful with. I would draw two very different categories here:

1. Depending on symlinks in order for zig to work, or even for some optional tooling to work. I veto this every time someone proposes it or makes a pull request that accidentally does this.
2. Creating general-purpose tooling that operates correctly with symlinks when they are used. This is the category we are in. Notice that to take advantage of this feature, you would not need symlinks. You could copy-paste folders instead, for example, or just have an extra version control checkout. The zig process isn't ever the party that would create a symlink. But if you wanted symlinks - which it seems like you are at least subconsciously expressing 2 comments above - in your workflow, then you can add them to your workflow, by modifying your development environment to support that desired workflow.

Final note, for the Windows use case specifically it would be nice to see some IDE integrations, for example, a UI that lets you quickly select a dependency and start patching it locally using this mechanism.

[scottredig]

Good distinction of categories. No further notes.

[mnemnion]

has ".zig-cache" dir with override added to source,

Note that this will never be the case. The .zig-cache directory must be excluded from source control, and must not be included in a package.

Must as in "Zig must return error if this will be the case" or must as in "Authors must refrain from this, but no hard checks"?

My interpretation of this is "zig will always treat .zig_cache as ephemeral, and will never exhibit behavior which requires the cache to be in source control, for any reason", for what that's worth.