Post-build script execution

[vosen]

Currently, cargo execute scrips before the build starts with the build field. I propose renaming build to pre_build and adding post_build (which would run after every successful build). It's useful for general postprocessing: running executable packers, zipping files, copying stuff, logging, etc.

[alexcrichton]

This will likely happen as part of an implementation of cargo install, but cargo build is not meant to be a general purpose build system for these sorts of applications.

[DiamondLovesYou]

What about integration tests? For example, PNaCl modules can be run on the host machine, but they have to be translated into a NaCl beforehand and then run with sel_ldr (from the NaCl SDK).

[jan-hudec]

@DiamondLovesYou, would https://github.com/rust-lang/cargo/issues/1411 work for the tests?

[caitp]

A "post_build" script would be useful for things like creating a MacOS bundle, rather than having a separate script to do that work separately. "install" could do this too, but maybe not as useful for testing there?

[vvanders]

+1 For rust projects that linked dynamically from other languages being able to deploy(move) the .so after a successful build would be nice.

[tbelaire]

This would be useful for running

arm-none-eabi-objcopy -O binary ${TARGET}.elf ${TARGET}.gba
gbafix ${TARGET}.gba

To fix up the headers and such after building my project.

[Boscop]

+1 for post build scripts

[mkroman]

This will likely happen as part of an implementation of cargo install, but cargo build is not meant to be a general purpose build system for these sorts of applications.

It doesn't have to mean that cargo will be a general purpose build system. An example where a post-build action could be used for cargo is triggering a build of a crate in a sub-directory once the main crate has been built.

Crate B depends on crate A, and is in a sub-directory of crate A, and it automatically gets built every time crate A has been built.

[Boscop]

But what if crate B also depends on sub crates C and D? Shouldn't the build of dependencies be caused by building the crate that depends on them in a "by need" manner? I would say the post build command should not involve building any related crates or calling cargo (can be easily enforced), only for post processing of the lib/exe that was just built. To keep it compartmentalized / clean.

[JinShil]

This would be useful for embedded systems programming. I often run arm-none-eabi-size to verify my binary size with each build, and occasionally other utilities from binutils to verify symbol tables, etc...

[steveklabnik]

Don't forget that there's nothing that says you have to use a non-Rust solution for these kinds of things; any "cargo-foo" executable on your $PATH works as "cargo foo", and can be written in any language, including Rust. Your custom command could invoke "cargo build" as a step first and then do anything.

On Oct 29, 2016, 13:35 -0400, Mike notifications@github.com, wrote:

This would be useful for embedded systems programming. I often run arm-none-eabi-size to verify my binary size with each build, and occasionally other utilities from binutils to verify symbol tables, etc...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (https://github.com/rust-lang/cargo/issues/545#issuecomment-257104800), or mute the thread (https://github.com/notifications/unsubscribe-auth/AABsimputy3ucYbBxCoVRqmOdAF8GQPnks5q44P0gaJpZM4CgUcP).

[JinShil]

Don't forget that there's nothing that says you have to use a non-Rust solution for these kinds of things; any "cargo-foo" executable on your $PATH works as "cargo foo", and can be written in any language, including Rust. Your custom command could invoke "cargo build" as a step first and then do anything.

Yes, that is what I, and probably others, are already doing. It would just be more convenient to integrate this into the already familiar cargo build.

[ssokolow]

Not to mention that, in my experience, if you try to enforce your will too strongly, you start to get behaviours similar to what has been standard in the home inkjet printer market for well over a decade now.

(Where the first thing you see when you open the box is a slip that warns you, in big letters, to never plug the printer in until after you've installed the software off their CD, because Microsoft's automated driver installation workflow doesn't allow them to install their fancy, tray-resident, bloated, talking print drivers and their official position is that, if Microsoft's installer has had a chance to run, your OS is now broken and must be repaired first.)

Better to have an official way to run a post-build script than to see everyone and their dog reinventing broken wheels... especially since, as a Stubborn Person™, I can easily foresee these horrors:

• Crates which bail out early and insist that, even if they're being used as a dependency, you must run their own special cargo mybuild wrapper instead of cargo build (possibly not as an inherent thing, but as a result of depending on something else by the same author which has that build requirement).
• Developers recognizing the ugliness of the former situation and producing a "better" solution by hacking (subtly buggy) post-builds onto cargo build using pre-build scripts which leave something running in the background to trigger the post-build using inotify or equivalent.

(Also, keep in mind that, as Rust sees more corporate uptake, we'll see more "I'm too busy to learn the 'proper' way, so I'll just hack something in" WTFs. That's just a basic rule of programming in general.)

[Kixunil]

In embedded development it's useful to run objcopy to produce hex file or something similar. Adding this to post-build script would be much cleaner than reinventing wheel with my own shell script.

[yasammez]

This would be useful for stuff like

SignTool sign /v /s PrivateCertStore /n MyTestCrt /t http://timestamp.verisign.com/scripts/timstamp.dll driver.sys

when developing Windows drivers in Testmode.

[fschutt]

This would be useful for packaging debian applications, especially when combined with configuration Like a configuration deploy which builds the crate in release mode, then creates the folders needed for a package (control files, etc.) and runs dpkg on it.

[kornelski]

cargo install touches files outside the current directory, so it doesn't feel like a proper post-build step. If I want to strip the built binary, would I have to work on a binary already copied to /bin?

[Boscop]

Why not do this: If there is a postbuild.rs file, it gets executed after build. (similar to the build.rs file before.) It would make sense to execute the script also if the build failed. The build status can be passed to it in an env var.

[Kixunil]

@Boscop I like that idea but I worry that executing another binary every compilation might slow down iterative development. Maybe using postbuild-success.rs for only success, postbuild-failure.rs for only failure and postbuild.rs for both?

[Boscop]

I think there won't really be a slowdown compared to having different scripts for success/failure because if your postbuid.rs script checks the build status first and then returns if it's not success/failure, it won't really slow things down. The postbuild.rs script won't have to be recompiled every time, only when it changes. So calling it will just incur the cost of starting an executable that immediately returns. Having one postbuild.rs script follows the KISS principle, most projects won't even have a postbuild.rs script, and for those that will have one, I think having one script fulfills their needs. If they notice a slowdown, we can always introduce the option to replace the postbuild.rs script by the other two later. But let's first have just the postbuild.rs and see if that's enough.

[Kixunil]

That sounds reasonable.

[tanis2000]

@Boscop idea of the postbuild.rs script sounds like a very good way of managing post build actions. Even though the general opinion seems to be split between this and avoiding to turn cargo into a general purpose make system, I feel that this won't hurt.

[JoeyAcc]

Not that I'm advocating for it per se, but Cargo already is a general purpose build system due to the cargo plugins (e.g. cargo tree) being able to do whatever they want. That that is merely a user-friendly(ish) facade for "random turing-complete script" doesn't diminish that in any way.

So the real question becomes: How ergonomic do we want to make this general-purpose build system?

[Boscop]

Cargo is not a general purpose build system in the sense that e.g. cmake is. But I see no reason why cargo can't be the best build system for Rust projects, and having post-build script execution would remove the need to wrap cargo build into another build system for projects that have post-build steps, which occurs more often than not.

[JoeyAcc]

Cargo is more restricted than e.g. CMake in the sense that there are assumptions about e.g. where to download dependencies, that each such dependency is a rust crate etc.

That doesn't mean it's not general purpose, since again each cargo "plugin" (AFAIK just some binary that masquerades as a part of Cargo) can do whatever it wants to, including downloading arbitrary projects (including non-rust ones), and mangling them in arbitrary ways, including building them.

That is why I view this as a discussion about ergonomicity rather than capability.

[ssokolow]

That is why I view this as a discussion about ergonomicity rather than capability.

Agreed. I won't restate my previous comment, but I highly recommend that anyone who hasn't seen/doesn't remember it should scroll up and read it.

There are a couple of ways in which I can very easily imagine people who "just need to get work done" foisting a a monstrosity of a bodge on the rest of us.

[matklad]

Just to make sure folks are aware, there are at least two Rust projects which make executing stuff in addition to cargo build simpler:

https://github.com/casey/just https://sagiegurari.github.io/cargo-make/

[JoeyAcc]

Especially cargo-make seems, at least for now, like the way to go if post-build steps are required. The utility name is apt, as it looks like a cross-platform version of make with much more intuitive "Makefile" syntax, since makefiles are just TOML files.

While that is certainly a workable solution for a lot of projects, I still have to wonder if long-term we want to incorporate that kind of functionality directly into Cargo.

[ssokolow]

The problem with cargo-make is that cargo build on a package isn't aware of any cargo-make or just scripting in the dependencies, so it's only a solution for --bin crates, not library crates.

For example, suppose you want to build a Rust binary that depends on a cdylib-based dlopened plugin for one of its other dependencies (or just wants to automate installing some plugins via features) and that cdylib needs to do some post-processing on the generated .so/.dll/.dylib file.

The only solution, currently, is to manually build the cdylib separately, using cargo-make or what have you, and then copy it into the depending project.

[JoeyAcc]

Couldn't a build dependency on cargo-make in the cdylib project (assuming it's a Cargo project) help with that, even when building a binary that depends on that cdylib?

Even though Cargo may not be explicitly aware of cargo-make, it know does know about transitive build dependencies, doesn't it?

[ssokolow]

How would that help to run something after the dependency is compiled without either forcing all downstream dependencies to also explicitly use/call cargo-make for/in their build automation or resorting to the ugly hackery I worry about where people fire off an inotify-based "watch for build completion" daemon in the pre-build script?

[JoeyAcc]

After the dependency is compiled this solution wouldn't do a thing of course. But if the dependency is a Cargo project, then Cargo can build it however specified in its Cargo.toml.

So this wouldn't help with e.g. pre-existing native (C/C++) dependencies, but it could help with some Cargo projects I think.

[ssokolow]

And the whole point I've been trying to make is that, in certain contexts, there are tasks which Cargo.toml doesn't understand which must take place after the build process.

(eg. modifications to the built ELF/Mach-O/PE files where include! and friends aren't applicable because the required format for the modifications is defined by some entity outside the developer's control, such as the operating system's loader.)

To summarize my first comment, if you don't accomodate these enough, you'll get horrendous hacks from people who care more about getting work done than tying themselves in knots to make the perfect codebase. (Such as the example I mentioned of firing off something in build.rs which sits around in the background, waiting for the build artifact to appear so it can manipulate it further.)

[kornelski]

RFC for build system integration may be relevant to this.

[Boscop]

What would be the best way to do post build script execution by starting a process that waits in the background; should that process use the notify crate to wait for a write event on target/release/<package name>.exe (for executables on windows)?

Would it make sense to publish a generic post build task runner tool, installable as a cargo binary?

[Kixunil]

@Boscop sounds very hacky to me. Anyone wanting to implement it would spend their time better by writing PR agains Cargo.

[SoniEx2]

Need to cat a zip to a bin. This would be useful, even if restricted to application/binary builds.

[gnzlbg]

I need to do something with my binary after I build it and before it is installed.

Without a post build script cargo install installs the wrong thing.

[FredrikAleksander]

In one of my projects, I need to generate a symbol map after building, and it is pretty annoying this cannot be done via Cargo. My current solution is using gnu make instead of Cargo, and have make call cargo when it needs to build rust. This is a really hacky solution, and it would be easily solved with a post build script

[dancrossnyc]

+1 on the need for this or something very like it. I need to run objcopy on a binary after build.

[timburks]

I'm also using rust to build macOS apps, currently by calling cargo from a shell script.

A suggestion that I haven't seen yet is to name the post-build script final.rs and run it after all builds, perhaps with environment variables to pass build status. This seems very non-disruptive to current usage.

[package]
# ...
build = "build.rs"
final = "final.rs"

[kornelski]

cc #3483

[antekone]

A post-build step would be very useful in environments which require Cargo to build the project, but don't support setting post-build action by themselves, i.e. Intellij IDEA with Rust plugin.

It's possible to create bash scripts that simply invoke cargo build and then perform selected action if return code was 0, and then add this script as a launch configuration, but still it would be very convenient if selected action would be executed directly after each build of the project.

Example of such post-build command is scp'ing the generated executable to a different machine, which contains large set of sample files that are impossible to transfer to the development machine.

[nox]

Note that with the advent of pipelined compilation, there is also a need for a post-build-but-still-produces-artifacts-required-for-linking scripts.

For example, the script crate in Servo depends on mozjs, which depends on mozjs_sys, which invokes make to build the JavaScript engine from Firefox, SpiderMonkey.

For obvious reasons, the script crate needs the Rust bindings for SpiderMonkey, but its compilation can start before SpiderMonkey itself finishes building. Unfortunately, as SpiderMonkey is built as part of mozjs_sys' build script, the pipeline is not as good as it could be.

[ianrrees]

I'm a new-to-Rust embedded developer, currently tinkering with a little ARM M0+ so there isn't much use in the ELF that cargo build currently generates. This proposal seems like it would be helpful - cargo-binutils is great, cargo objcopy --release -- -O binary firmware.bin produces my desired output at this stage. But, it seems awkward to require a different command from a simple cargo build to get the desired output.

Since this ticket is still open, I assume this is still up for consideration?

[anderejd]

In the .NET world we have dotnet publish in addition to the regular dotnet build, perhaps an official cargo publish would be a less controversial command to allow post-build scripts for?

EDIT: I just realized my brain fart above. cargo publish is taken for uploading to crates.io. Perhaps calling it cargo package would work?

[kornelski]

npm has lots of scripts for various lifecycle events, including "prepublish"/"prepublishOnly", but use-cases for that are notably different from general post-build:

• Compiling CoffeeScript source code into JavaScript.
• Creating minified versions of JavaScript source code.
• Fetching remote resources that your package will use.

but there's no "CoffeeRust" yet. I thought that maybe I could use a prepublish hook to pre-generate some lookup tables (to avoid building them in build.rs), but I couldn't either: I need them in cargo test too.

cargo publish can't be used for publishing compiled binaries, so tasks like stripping binaries and creating macOS framework and app bundles wouldn't be satisfied by a prepublish hook.

[steveklabnik]

@ianrrees the general idea has been stuck in "RFC hell" for a while; I am doing the same thing (roughly) as you, but using https://github.com/matklad/cargo-xtask (which links to one of the proposals for stuff in this space) as a polyfill in the meantime.

[ianrrees]

@steveklabnik is there an RFC for post-build scripts? I read through RFC 2136 which @kornelski mentioned in the thread above, and didn't get the impression that this proposed change would conflict. That also made me wonder whether this feature would be big enough to warrant an RFC, assuming that the existing build script functionality weren't broken.

[steveklabnik]

If there is one, it's not one that's well-known to me.

I am not on the Cargo team, but IMHO, this does deserve an RFC, because even if it is a small feature on its own, it would need to be coherently designed alongside any other desired lifecycle hooks.