Post-build script execution

[vosen]

Currently, cargo execute scrips before the build starts with the build field. I propose renaming build to pre_build and adding post_build (which would run after every successful build). It's useful for general postprocessing: running executable packers, zipping files, copying stuff, logging, etc.

[octoape]

Share my idea

• Create a post_build project in the project root directory cargo new post_build
• Some post-build logic in post_build/src/main.rs

  fn main() {
  // Your post-build logic
  }

• Execute the following commands in the project root directory to build the project cargo build && cd post_build && cargo run && cd ..

[sagiegurari]

that won't work for libraries that you include in your project

[nickgerace]

@mumashanren I think you are nearing the cargo-xtask pattern, which is a common interim-replacement for the would-be feature corresponding to this issue.

• https://github.com/matklad/cargo-xtask
• https://github.com/nickgerace/cargo-xtask-example

[octoape]

that won't work for libraries that you include in your project

Yes, this cannot solve the problem that depends on the cargo lifecycle.

[octoape]

@mumashanren I think you are nearing the cargo-xtask pattern, which is a common interim-replacement for the would-be feature corresponding to this issue.

• https://github.com/matklad/cargo-xtask
• https://github.com/nickgerace/cargo-xtask-example

A good idea can solve my problem. Thanks!

[joshtriplett]

Following up on this: as noted in #10430, a mechanism for post-build scripts would additionally need to take cargo's multitarget feature into account, allowing for the possibility of running after building binaries for multiple targets.

[Kixunil]

multitarget link for the lazy :)

[mokeyish]

How to set post-build script now? I want to debug rust applicaiton without root in linux, it need to setcap before lauching otherwise the error of Permission denied will throw.

[sweihub]

I am looking for this, it seems NO? So the workaound to write a build.sh

cargo build
echo "Hello, post build greeting!"

[marziply]

I would find this useful for tarballing the final executable to automatically pipe into a Docker image. For microservice monoliths, like the one I am currently working on, this would be very handy. I will delegate this to cargo-xtask instead for now but having this available directly in cargo would be a great addition in my opinion.

[ssokolow]

I think the best/most compelling use-case for Cargo-supported post-build scripting is stuff where "If this post-build step isn't done, the resulting binary will be broken. Full Stop. Not a supported configuration. Good day sir."

In such a situation:

1. Post-build script support is both necessary for cargo run to work and in line with the Cargo philosophy of being Cargo, not a hostile-to-new-users/developers forest of autotools/CMake/etc. hackery where you have to pay close attention to the documentation to get a working build.
2. It eliminates one source of irritating "Humans gonna human" bugs that get closed as something like WONTFIX NOTABUG in Bugzilla parlance.

[axos88]

This could also be a solution for generating output artifacts other than the binary itself.

The build script can be used to generate inputs to the compilation step. But if I want to say generate jsonschemas, typescript bindings, protobuf definitions based on some of my struct definitions in the code, I'm forced to use external tools. ts_rs basically generates and injects the file contents as strings and suggests to use tests(!!) To write the strings out to actual files.

[dlight]

@axos88 you can generate them on build.rs (that is, before the build) and configure to re-run on file changes. The only problem with that is that it will generate even if the build fails.

[axos88]

@dlight In theory, for simple use cases, yes. For more advanced things, not really. For example in case the build.rs actually generates code as input for the build process or in case some of the code you would want to generate bindings for are actually generated by macros, then not so much.

[axos88]

And @ssokolow i believe this use cases applies. Without the correct bindings generated, one wouldn't be able to interface correctly to the generated binary so it couldn't fulfill it's purpose

[Kixunil]

@axos88 I actually have such library and I came to conclusion that using an external tool for artifacts not required by build really is better. Some advantages:

• Trivial cross-compilation - just run the same tool for any platform!
• Can reduce build time if the tool is pre-built, may even make sense to build it with --release
• Integrates into (debian) packaging nicely
• Doesn't bother people who don't need those artifacts

[axos88]

@Kixunil I'm not sure your example applies. You are generating rust code from other inputs, no? I'm referring to generating other outputs FROM rust code. For example in glue code for FFI and or serialization of API data structures for typescript, java, python, brainfuck, whatever.

[Kixunil]

I do both. configure_me_codegen generates Rust from Toml, cfg_me generates man page from the same Toml.

[melvyn2]

Some of my tests require the test binary to be signed on macOS (for entitlements & such). Having to manually sign the test bins and re-run them outside of cargo makes proper tests a lot more annoying than they need to be (especially considering I could easily sign them with apple-codesign in the hypothetical postbuild.rs).

[dlight]

Currently my idea to workaround this issue is to make a build.rs that fails compilation except if a given environment variable is set, with a message saying it only builds with cargo xbuild.

Then, add aliases cargo xbuild and cargo xrun on .cargo/config.toml in the project that will work just like cargo build and cargo run, but setting the corresponding environment variable and running the postbuild script (in the cargo xrun case it's more complicated because it needs to run postbuild after building but before running the program, but this is doable)

This is very messy but it's the only way to absolutely prevent building without running a postbuild script. Having postbuild.rs incorporated in Cargo proper would be greatly appreciated.

[e792a8]

My idea is to reuse build.rs.

For example, in build.rs we write:

// Tell cargo we have post-build actions ...
println!("cargo:post-build");

// ... so that cargo runs the same build.rs again on post-build stage,
// but with an envvar CARGO_STAGE_POSTBUILD set
if std::env::var("CARGO_STAGE_POSTBUILD").is_ok() {
    // post-build actions
} else {
    // pre-build actions
}

[ssokolow]

It'd have to be enabled via a field in Cargo.toml or it would break every current usage of build.rs which doesn't check CARGO_STAGE_POSTBUILD, and, if you're doing that anyway, it'd be cleaner to have a separate file rather like main.rs vs. lib.rs as opposed to some kind of hypothetical #[cfg(bin)] vs. #[cfg(lib)].

[e792a8]

it would break every current usage of build.rs which doesn't check CARGO_STAGE_POSTBUILD

I have considered that - if we do not println!("cargo:post-build") then cargo should not run it post-build.

it'd be cleaner to have a separate file rather like main.rs vs. lib.rs ...

Well, I think pre-build and post-build are both things about "build", so combining them in a single file named build.rs seems not that bad. But having extra files may make the source tree more complicated. Just like the System V init scripts, where a single script defines a service and "start" "stop" actions are handled in different branches.

[m4heshd]

It's INSANE how this level of a requirement is entirely ignored by devs. My god. What is so evil about implementing such a useful feature for developers? This is what makes Rust devs have that alien package.json in their Rust project tree. (You all know what I mean)

PS: I'm currently working on anger management. 😁

[Qix-]

@m4heshd A few things

1. OSS is done in spare time for the most part, so the velocity of these conversations is generally slower than it would be in a corporate environment.
2. Rust puts great emphasis on doing things correctly, or at least, as correctly as possible. This further slows the velocity of these discussions.
3. Decisions about Rust are democratic to an extent, meaning that no single person can make a unilateral decision about a feature. It must be a collective effort, and everyone must have the ability to give input. This even further slows the velocity of these discussions.

There is some merit to the discussion against having a post-build. I do not agree with some of the opposition, but there are problems and edge cases that would have to be solved.

Certainly coming here and demanding things with a rude tone will not help the situation any. In fact, in my experience, it usually sets the conversation back, perhaps indefinitely🙃

[m4heshd]

OSS is done in spare time for the most part

Been doing that for 2 decades and you and I have very different views on that. Doing open-source work in spare time does not mean that they're free of responsibility. For me, it was always been about how I would make things easier for the people who use the stuff I create. That was always the priority. You can hold me accountable for this statement because my footprint in open-source history is very public. So remember, there's always something new to learn. Somebody's way is always not THE WAY. Sorry for ruining your "teachable moment".

Certainly coming here and demanding things with a rude tone

"Rude tone", oh come on. 😆 Also I don't have to demand anything. We're problem solvers by nature. There are hundred-thousand different ways to achieve what I want. But inconsistency just hurts. Not to mention it's an ugly thing to have. What my "rude tone" emphasized was how ridiculous the reasoning behind why NOT to have this feature implemented. This issue was created in 2014. I'm yet to see the so-called "democracy" in this.

Well, going forward, my response to this issue and your response to it would be my reasoning as to "why I had to" when it comes to Rust.

[ssokolow]

Been doing that for 2 decades and you and I have very different views on that.

I've been around since 2013 and, if there's one thing that's defined Rust's evolution all along, it's "RESO POSTPONED: Not enough manpower and we need to prioritize the stuff that's most difficult to work around".

https://lib.rs/ partly exists because it's been taking years and years to rebuild https://crates.io/ to support more SEO-friendly server-side rendering.

Many RFCs for the language itself have been "closed as postponed because we want to focus on shrinking the existing backlog".

Cargo is no different, and, as much as I personally want post-build scripts, lack of them is simple to work around with a wrapper.

"Rude tone", oh come on.

"It's INSANE how this level of a requirement is entirely ignored by devs." is rude. They're not ignoring it. They just triaged designing something they'd be willing to support in perpetuity as way way down a TODO list sorted by how urgently people need a first-party solution.

You're insulting their character and motivations with that assumption.

[omarabid]

It's INSANE how this level of a requirement is entirely ignored by devs. My god. What is so evil about implementing such a useful feature for developers? This is what makes Rust devs have that alien package.json in their Rust project tree. (You all know what I mean)

Calm down.

[omarabid]

@m4heshd A few things

1. OSS is done in spare time for the most part, so the velocity of these conversations is generally slower than it would be in a corporate environment.

2. Rust puts great emphasis on doing things _correctly_, or at least, as correctly as possible. This further slows the velocity of these discussions.

3. Decisions about Rust are democratic to an extent, meaning that no single person can make a unilateral decision about a feature. It must be a collective effort, and everyone must have the ability to give input. This even further slows the velocity of these discussions.

There is some merit to the discussion against having a post-build. I do not agree with some of the opposition, but there are problems and edge cases that would have to be solved.

Certainly coming here and demanding things with a rude tone will not help the situation any. In fact, in my experience, it usually sets the conversation back, perhaps indefinitelyupside_down_face

This is going to be an ongoing problem as Rust gets exposed to more people, especially developers coming from the JavaScript/NPM background where the mentality is to add the feature now, deal with the consequences later. I appreciate that the current Rust community stands against this trend.

[omarabid]

This could also be a solution for generating output artifacts other than the binary itself.

The build script can be used to generate inputs to the compilation step. But if I want to say generate jsonschemas, typescript bindings, protobuf definitions based on some of my struct definitions in the code, I'm forced to use external tools. ts_rs basically generates and injects the file contents as strings and suggests to use tests(!!) To write the strings out to actual files.

I have the exact use case. I have a library that has a GraphQL component. It works on its own when being called from a server. However, for simplicity, it would be preferable to have something like build.rs which could generate the .graphql schemas. These artifacts being generated at build, would make them usable before the deployment of the library itself.

At the moment, this can be solved by adding a main.rs file. Essentially rendering the library a binary, and then running the binary to achieve the same effect. This wouldn't apply, however, if you need these artifacts to run tests. In this case, a post-build script will come in really handy.

[sndnvaps]

it's better to have the new func of post-build in build.rs ...make it easy to pack the app to zip or tar ball.

[jan-hudec]

There is plenty of generic tools for building all sorts of packages and installing software in various ways etc., so I think cargo should not try to be everything for everybody. It should focus on the use-cases needed by library crates, because those will be built as dependencies of other projects and cargo needs to be able to handle that completely. But the packaging and distribution (deb, rpm, container, snap, app bundle, winget etc.) can be left to wrappers and extensions using the standard tools for those.

[Qix-]

The problem with "standard tools for those" is that there are not standard tools for those. The era of having non-portable, extensively quirky and problematic tooling should end at some point. How much of that Cargo can and should handle is of concern, of course.

For example, Makefiles are non-portable, shell scripts are very much non-portable, etc. Even the C/C++ world has CMake which is mostly portable (at least, it gives you the option of writing portable configs) and it comes both with a testing framework (CTest) and a packaging framework (CPack). If you're eagar enough it can also generate multiple targets with a single command, something Cargo cannot do right now and something which is somewhat of a headache for e.g. embedded and low-level software development.

[jan-hudec]

They are standard tools for their purposes. Yes, dpkg-buildpackage will only run on Linux, whatever builds MacOS packages will only run on MacOS, most tools packaging for Windows will only run on Windows, and embedded usually has its special image-building machinery like bitbake (yocto). But trying to reimplement them would mean forever playing catch-up and there's not enough manpower for it anyway. And adding wrapper to call them will not do anything about the portability.

And if you want something for generic additional tasks, make might be non-portable, but there is gradle, buck or bezel that all are reasonably portable and generic. Buck is even being rewritten in Rust.

[Qix-]

I'm not referring to the individual platform-specific packagers - those are implicitly platform-specific.

I am talking about the task runners. No, sorry, in many, many cases that I won't get into here, and for many, many reasons, neither Gradle, Buck nor Bazel are acceptable tools for the job - especially when Cargo is involved. It doesn't matter what they're written in.

[gdennie]

cargo_make seems to implement a flexible build tool...

[Qix-]

It's not a standard thing with cargo and requires a third-party installation (via cargo install cargo-make or something). Yes those tools exist, but getting things to run as part of a dependency chain or something whereby standard Cargo processes are happening, cargo make doesn't help.

To put it another way, if cargo make was the standard way cargo worked with projects and would run them even in nested dependency trees, that'd be a different ballgame. I'm not advocating for that to be the case, though. But it would solve the issue directly.

[leryn1122]

I do think it's neccessary to provide an official hook as post-build scripts, rather then

• write a glue-lang/shell script with no platform portability
• third party implemented cargo-make / cargo-blablabla, it's not a standard process

Two relevant cases I got are,

• manually apple-codesign executables with entitlements after every building, it makes unit test quite exhausting
• package my app as a desired bundle or tarball for distro

Possible implemention could be:

• println!("cargo:post-build=/path/to/post-build.rs") in build.rs
• post-build.rs under project root, as equivalent as build.rs

[gdennie]

Perhaps support for the occurrence of scripts throughout the build process couldbe accommodated an enhanced with an in memory database or file system to facilitate communication or resource sharing between scripts...

pre_build pre_compile pre_link post_build

I can't see where such a thing wouldn't be immediately valuable.

However, isn't there an existing need to sandbox build scripts somehow such as by restricting resource access through a monitoring agent.

[epage]

For myself, I feel like post-build scripts would be an incremental but incomplete improvement that is a dead-end. What we really need is a build orchestrator above cargo for solving these kind of use cases. For example, if you want a universal binary, you need a post-build script that combines the result of two targets.

I wrote more of my thoughts at https://epage.github.io/blog/2023/08/are-we-gui-build-yet/

[nvzqz]

@epage could you please give examples of prior art for such an orchestrator? Do you mean meta build systems like Nix/Bazel/Buck?

[dlight]

@epage an external build system can run a command after the whole build completed, but it can't run after the build of each crate, so it doesn't completely replace a Cargo integration for postbuild scripts. (one use case is to patch each object file after they are built, but before linking)

@nvzqz perhaps he is referring to something like just or cargo-make

[dlight]

I just found cargo-post which seems like a nice workaround.

However it always run the post_build.rs on every successful build (even if there is nothing to do), rather than running only when the output artifact changes.

[epage]

@epage an external build system can run a command after the whole build completed, but it can't run after the build of each crate, so it doesn't completely replace a Cargo integration for postbuild scripts. (one use case is to patch each object file after they are built, but before linking)

All the use cases I've seen are for final artifacts only (bins or SOs). what do you want to do to the rlibs?

[dlight]

@epage right now, indeed, I don't!

But here's an use case: what if I wanted to replace all instances of a certain function call in a crate for something else? (be it a call to another function, or even something more custom)

One idea to implement this is with a custom MIR -> MIR compiler pass, but I'm not sure this is even available and if it is, it's certainly nightly-only with not prospects for stabilization (I guess it would require stable MIR, at least)

But we could operate at a lower level. There are already lower level tools like wasm-snip that replace function bodies with no-ops, to save space if the function is only called by dead code that wasn't eliminated by the optimizer. But that operates on the whole artifact.

But if such a tool could operate on individual crates, it could allow for more fine grained manipulations, for other purposes. Something like: I want that all allocations done in this crate to be replaced by a panic, but I don't want to instrument the code or anything; I could just add a postbuild script to this crate to modify the rlib (or dylib or whatever).

[Kixunil]

After a long time I came to conclusion that make-like software may be really the best for this and not cargo however it needs information about which binaries are built from which sources. It already has an incomplete information in various *.d files. Specifically, the root .d file does not list the dependencies - only the source files from the project itself. I haven't figured out if this is fixable. Without this, incremental development is very messy.

[ssokolow]

After a long time I came to conclusion that make-like software may be really the best for this and not cargo

For the complex cases, perhaps, but it doesn't feel reasonable to require every Cargo integration (eg. VSCode plugins) to add support for inserting a project-specific wrapper around Cargo calls just so you can do stuff to the final artifact like non-automatic codesigning which may need to happen on some targets before something like cargo test will work or renaming libraries for a plugin/extension API that requires it. (That's still a problem, right? I didn't miss a fix for that?)

That feels like it's being disproportionately unfair to, for example, platforms which aren't self-hosting and require either emulation or integration with a remote device such as mobile and bare metal targets.

[gdennie]

Scripts invoked by hooks and provided pre-compiled libraries offering API into the build process can perform all manner of inspection and augmentation of the build throughout. In fact, too much perhaps; consequently, a sandbox, permission system, or a constrained API seems equally essential to manage the scope, at least for scripts of external dependencies. Ultimately, restricting scripts to a predetermined collection of domain specific libraries could very efficient, manage permissions of such scripts, and yet offer an extensive API into the build process.

[ssokolow]

Scripts invoked per their hook and provided domain specific precompiled libraries offer such thingsas an API intothe build process should be sufficient for all manner of mischief in any stage of the build. In fact, too much mischief; consequently, a sandbox or permission system seems equally essential to limit their scope, at least for scripts in dependencies. Ultimately, restricting scripts to a predetermined collection of domain specific libraries should reduce the required compilation effort, provide a means of managing their scope, and offer an extensive API into the build process.

A way to enable that was already proposed and accepted... it's just waiting for someone to implement it:

https://github.com/rust-lang/compiler-team/issues/475

...though it'll probably be for proc macros first since they're much more likely to be pure.

See also https://github.com/rust-lang/cargo/issues/5720

[Kixunil]

@ssokolow oh, sure, I'm not against it. I just happen to have that more complex case and it'd be nice being able to get the correct list of dependencies. (Not a big deal since that is only needed for edge cases now.) Or better said, for my case post-build scripts (to generate man pages, completions...) are not very suitable while for generating binaries they are. So ideally we should have support for both.

You make good points about tests & stuff.