ability to depend on pure zig modules without running any build.zig logic

[andrewrk]

Extracted from #14265.

Terminology clarification: #14307

For many zig modules, advanced build system features are not needed, such as generating source files, compiling C code, or accepting build options. In such cases, build.zig is overkill. All that is really needed is the .zig source files to be available to the project which has the dependency on this one.

For this I propose the package to be organized like this:

• build.zig - may or may not exist. Perhaps the package still has a test harness to be run, or has development dependencies.
• build.zig.zon - may or may not exist.

Meanwhile the package depending on it will do something like this:

build.zig.zon

.{
    .name = "clap",
    .url = "...",
    .hash = "...",
}

build.zig

// "clap" here references the dependency.name field of the build.zig.ini file.
const exe = b.addExecutable("app", "src/main.zig");
exe.addModuleFromDependencyPath(
    // name to be used by @import
    "clap",
    // module root directory, relative to clap dependency build root
    ".",
    // module root source file, relative to module root directory
    "clap.zig",
);

Note that this allows a project to add a dependency based on any arbitrary directory of files without the dependee being aware of it being used that way.

When a module is added this way, no build.zig logic is executed. Note that this could be problematic, such as in the case of a dependency introducing a new generated file as part of their package, or adding build options. To future-proof against this situation, instead of addModuleFromDependencyPath, build scripts should use b.dependency() which will execute the dependency's build logic.

See #14278 for depending on zig packages while respecting build.zig logic.

See #14286 for running build.zig logic in a sandbox.

[deflock]

The filename, build.zig.ini is intended to imply that it is an appendage to build.zig because that is exactly what it is. The real, actual file that signifies a zig package is a build.zig, and the existence of this extra file is bonus - it is for the case of declarative information that we want to expose without requiring execution of zig code.

Is this still relevant with this change?

[nektro]

since entry point and transitive dependencies are likely to be declared by the source dependency itself, imo it would make more sense for build.zig.toml to be required and to make addDependencyPackagePath() accept a name and a path to a build.zig.toml rather than a zig source file

[ikskuh]

since entry point and transitive dependencies are likely to be declared by the source dependency itself, imo it would make more sense for build.zig.toml to be required and to make addDependencyPackagePath() accept a name and a path to a build.zig.toml rather than a zig source file

An alternative to that is to use the usual build.zig.${ext} approach and encode the exported packages inside that file:

[project]
name="tiny-package-bundle-3"

[[package]]
name="leftpad"
root="src/leftpad.zig"

[[package]]
name="rightpad"
root="src/rightpad.zig"
dependency=["stringhandler.stralloc"] # imports the package 'stralloc' from the dependency 'stringhandler'

[[dependency]]
name="stringhandler"
url="https://ohno.example.com"

So we can just use the dependency in the main build.zig:

pub fn build(b: *std.build.Builder) void {
   …
   exe.addPackage("tiny-leftpad", b.dependency("tiny-package-bundle-3").package("rightpad"));
   …
}

This will allow zig-only projects to import other projects, without having to use build.zig logic for the project. Also a project can export more than one package.

Use case for this: Zig Embedded Group can provide family support bundles, for example "AVR" as a project, that exports all the AVR family microcontroller support packages (which would be a huge load of small packages like atmega328p, atmega32, atmega8515, …)

[david-vanderson]

Can we do this recursively? So an app (with build logic) that uses a pure zig dependency A that uses a pure zig dependency B, and neither A nor B have build logic?

When developing library A, how do I import stuff from B without a build.zig?

[andrewrk]

That's a great point. It demonstrates that this idea only works if it can be encoded via purely the declarative file.

[leroycep]

It would be nice to just get access to the raw files. There already projects that have multiple Zig packages in a repository, and you just pass it a path to the file. What if the path to package was just exposed?

// "clap" here references the dependency.name field of the build.zig.ini file.
const exe = b.addExecutable("app", "src/main.zig");
exe.addPackage(.{
    .name = "clap",
    .source = .{ .path = b.pathJoin(&.{ b.packagePath("clap"), "clap.zig" })  },
});

Failing that, there is the @src().file pattern that could be used in conjunction with #14279 allow the build.zig script to create the Pkg for module's user. However, that defeats the purpose of this proposal and it feels like it should work more along the lines of adding a c dependency.

I personally would like to use something like b.packagePath("clap") to depend on a binary release. For example I depend on ZWO ASI's SDK, which only contains binary .a and .so files. While I could have some other system of getting the library files (like say, checking it into a git repository), it would also be nice to specify that in the build.zig.ini, and then add the include/library path using b.packagePath("asicamera").

(Actually, in this case I probably wouldn't link there because they don't version the download link. But I don't think this is the only case where a project depends on a binary file downloaded from somewhere else.)

[andrewrk]

I'm changing this to a proposal. I'm starting to get the impression that this may be a terrible mistake.

[leroycep]

Could you say what makes you think that? I can see fetching arbitrary archive files as perhaps distasteful; but without a centralized server there is only so much that can be done to prevent it. Especially if #14294 is implemented, any plugin could preprocess the file to make it work with the package format.

Anyway, I just want to know if there's a reason to specifically disallow treating fetched packages as another folder you have access to.

[david-vanderson]

I think the option to fetch arbitrary archives is still on the table, but should be considered separate from this issue. See https://github.com/ziglang/zig/issues/14488

For pure zig dependencies, I'm starting to agree with @andrewrk. If there's no build.zig, then somehow putting things into build.zig.zon would have to magically alter what you can import in zig files. Then for a given zig file the compiler would have to find the "right" build.zig.zon.

The alternative is mandating a build.zig file to be able to use any dependencies, which seems straightforward and likely in any case.

[andrewrk]

I am rejecting this proposal to avoid the problem of a package which needs to introduce build logic, but is unable to due to having dependents which would be broken by such a change. Similarly, a project may wish to move source files around, and should not need to worry about a dependent which has hard-coded the file path of the source files.

This decision could be reversed later; however I think the best approach at least for now is to reject this proposal.