Export Rust build environment for use by other buildpacks

[emk]

See #9. The theory here is that we might have a Ruby application that uses a gem implemented in Rust, and in order to compile the gem, the Ruby buildpack needs access to the Rust compiler.

To support this, we need to create an export file in our buildpack directory containing all the environment variables needed to run cargo and rustc.

Note that this still has some limitations: You almost certainly need to have a valid Cargo.toml in your project for the buildpack to detect and build before you can run the other buildpack that needs Rust.

• CC @binarycleric Feel free to take a look at this and let me know what you think.
• CC @schneems @elifoster Please let me know if this works for your use cases.

To try this, I think you can run heroku buildpacks:set https://github.com/emk/heroku-buildpack-rust#multirust_export and redeploy. But please set it back when you're done, because I'll delete the branch when I merge it.

[elifoster]

Requiring a Cargo.toml shouldn't be necessary and will fill projects that use Rust libraries with bloaty Rust stuff. Ideally all we should need is a RustConfig to specify the Rust version we want to use (or channel). Cargo will fail if it does not find the main boilerplate Rust stuff (src/lib.rs or src/main.rs I believe), because it actually needs something to build.

I would recommend checking if Cargo.toml is found, and if it is, then perform cargo build. If it isn't found, simply log that it was not found and continue.

[emk]

@elifoster I'm happy to try to support this use case, but if I do so, I'd like to do so in a "standard" fashion. Can anybody find me some docs on how Heroku's multi-buildpack support actually works?

Going by the example of the official Node.js buildpack, it looks like bin/detect should fail if there's no Cargo.toml. But even after reading the source code for bin/compile, I'm not convinced that I know what happens if there's no package.json.

Basically, I'm happy to implement this as long as we can find a good example of an official buildpack that does something like this.

[schneems]

If you're deploying Ruby, you're required to have a Gemfile for bin/detect. That's the minimum viable thing we can use to determine you're using Ruby. Now later in the compilation phase if we determine that you don't have a Gemfile.lock we print a warning and exit since we cannot continue.

If you try to deploy a node project without a package.json then detect will fail https://github.com/heroku/heroku-buildpack-nodejs/blob/master/bin/detect#L4. In that case the buildpack won't get run but if you're using multi-buildpack or heroku buildpacks:add then the next one will run. In that case bin/compile never executes.

So there is no way to deploy a Ruby app without a Gemfile and no way to deploy a Node app without a package.json. If those files don't exist the buildpacks will be skipped. If there are problems with those files or something else in the project then exceptions will be raised later by the buildpack to halt the execution.

[elifoster]

The difference between Rust and Heroku's officially supported languages is that Cargo actually requires there to be Rust source code in the project being built. With Bundler (for Ruby), you can execute its commands without having any source code present (well, not bundle exec, but that shouldn't be expected). So if you simply have an empty Gemfile, you should be able to successfully install Ruby on a Heroku dyno regardless of if there is any actual Ruby code in the project.

I have no experience with NodeJS so I won't speak for that.

You could require at the minimum the RustConfig file or the Cargo.toml file, since it appears RustConfig is currently optional.

[emk]

@schneems OK, based on what you said, it sounds like there's no precedent for this use case.

If I'm understanding things correctly, @elifoster wants to use heroku-buildpack-rust to set up a reasonable Rust environment, including things like rustup, cargo and the compiler toolchain. But any actual Rust code will be buried somewhere deep inside a gem, and will have its own Cargo.toml. This actually seems like a reasonable design. The only weird part is not having a top-level Rust project.

And as you say, there doesn't seem to be any precedent for this situation.

So maybe the answer is to have some explicit way of indicating:

1. Yes, this is a Rust project as far as bin/detect is concerned.
2. No, there wasn't supposed to be a Cargo.toml file, so just go ahead and set up the toolchain.

We could put this flag in RustConfig (as RUST_SKIP_BUILD=1 or whatever), but I'm trying to get rid of RustConfig, because it's an artifact of much older Rust and Heroku toolchains. It looks like buildpacks now have access to config variables. Would it be reasonable to use heroku config:set RUST_SKIP_BUILD=1 to control this behavior?

(If so, we could move VERSION into a RUST_CHANNEL config variable. That would seem pretty elegant to me, if that sort of thing is common in other buildpacks.)

[schneems]

There is some precedent here, as you mentioned you could have a blank Gemfile and Gemfile.lock or a relatively empty package.json. It's pretty common that you might want a specific version of Node to compile your assets in a Ruby app, but don't need to install any NPM packages, for that you would need to declare that you need the nodebuildpack as well as adding the package.json.

1. Yes, this is a Rust project as far as bin/detect is concerned.

Historically this is the presence of a file. it makes as much sense as setting an env var or any other kind of flag the user could set. It's nice that you check it into your source and if you were to deploy the app to another Heroku app then there's not another flag you would need to toggle.

Both Ruby and Node support a "default" version that gets installed if you don't explicitly specify. For node I actually think they have a version specifier, something like >= <version>.

Is it possible to have a Cargo.toml with no dependencies? Just empty, let's your buildpack know that yes it is Rust and to install the toolchain.

[emk]

Both Ruby and Node support a "default" version that gets installed if you don't explicitly specify.

We have this now! We default to the Rust stable channel if no version is specified.

Is it possible to have a Cargo.toml with no dependencies? Just empty, let's your buildpack know that yes it is Rust and to install the toolchain.

Not that I'm aware of. A Cargo.toml file describes a project to build, and not merely a set of dependencies to install. Cargo is very opinionated, and it will just happily assume a default directory layout and naming convention if you don't tell it anything. Even if we could get it to work with no source files, I wouldn't trust it to continue working unless the Cargo maintainers added test cases to ensure that it continued to do what we want.

I really don't like RustConfig, but I suppose we can keep it if there's no better way to do this. My previous plan was to use [metadata] in Cargo.toml to specify the Rust channel, but that won't work if we want to install tools without a Cargo.toml.

So if people really feel strongly that we should use a file as our "don't build any Rust app" indicator, we could do that. But I'm still not fond of RustConfig.

[elifoster]

Even if we could get it to work with no source files, I wouldn't trust it to continue working unless the Cargo maintainers added test cases to ensure that it continued to do what we want.

Agree.

I support replacing RustConfig with actual environment variables.

[emk]

I agree that there's going to be growing interest in writing Ruby gems in Rust and deploying apps which use those gems to Heroku.

Even though I'm not fond of RustConfig, maybe we should treat that as a separate issue. So let's put the flag variable in RustConfig, and use that to know when it's OK to not have a Cargo.toml. I may be able to find some time to do this over the weekend, but if somebody wants to send a PR, I'd be delighted to take a look!

See upthread for the proposed variable name, with a RUST_ prefix. I'm at least going to rename the variables in RustConfig to use better names, and only support VERSION for legacy deploys.

[emk]

OK, I've merged @elifoster's RUST_SKIP_BUILD support. Looks great! Now all we need to do is test this and update the README.md file with instructions on using these new features.

[emk]

OK, I fixed one minor regression, and I'm going to go ahead and merge this. Thank you for everybody who helped implement this great new feature, and please let me know whether it works for you!

[emk]

By the way, I'd like to add an automated test case for a Ruby application that includes a Rust-based gem, just to make sure that this new feature doesn't regress. (We already have automated tests for standalone Rust applications.)

Does anybody know of a gem on rubygems that uses Rust? Would anybody be interested in creating a rust_rust_hello that needs both a Rust and a Ruby buildpack to deploy to Heroku?

[elifoster]

Does anybody know of a gem on rubygems that uses Rust?

string-utility version >= 2.7.3 uses Rust for String#spacify and String#underscorify.

Would anybody be interested in creating a rust_rust_hello that needs both a Rust and a Ruby buildpack to deploy to Heroku?

Assuming you mean rust_ruby_hello, I could look into it :+1:

[emk]

Assuming you mean rust_ruby_hello, I could look into it

Erm, yes. Sorry. 🙂 I'd be happy to integrate it into test_buildpack. All I need is a GitHub repo which tests this case.

[schneems]

Thanks for all the work here!

BTW the Ruby buildpack uses https://github.com/heroku/hatchet for setting up apps and running tests and doing assertions (tests are written in Ruby) if you're interested.

[emk]

I've announced the recent updates on r/rust! The post is here. Thank you to everybody who made this possible!

@schneems Thank you for the link to hatchet! I'll keep that in mind if testing gets more complex that what I can easily do with a shell script. Or I'll eagerly welcome PRs, of course. :-) Does hatchet use Docker on Travis CI to simulate a Cedar-like environment? That's pretty important for a compiled language like Rust, I think.

[schneems]

You create a new user and give travis the API token for that user. It then actually creates real world heroku apps and deploys to them using git templates you provide. You can then do things like assert that the deploy worked or run heroku run echo $RUST_ENV_<var> and assert on the output.

The upside is that it simulates real world really well since it is actually deploying apps. The Ruby buildpack is pretty old, so it handles a TON of edge cases that we constantly have to test for. Here is our github repo with all the different templates https://github.com/sharpstone. The down side is that it can be brittle depending on how you're doing your assertions. I also think I've got some flags on my buildpack testing user to let the API token live longer than usual.

[emk]

OK, that's probably overkill for us right now but I'll keep it mind. :-)

We use the official heroku/cedar Docker image to simulate two builds: One from scratch, and one with a cache. It's not quite real cedar or a real deploy, but it should be fairly close.

[elifoster]

(#15)

Also it should probably be mentioned that Travis currently cannot handle installing more than a single language, unless you want to install it in the before_script phase.

[emk]

Thank you for the bug report! Could you please report this as a new issue?

Also, beware dynamically linked Rust. We don't currently copy in the *.so files for the rust runtime.