Including submodules (recursively) in the source tarballs corresponding to a release

[rob-p]

Hi!

This is a feature I've been wanting from GitHub forever, but I was wondering if it's possibly within the ecosystem of cargo-dist. Basically, when I tag a release and cargo-dist generates the relevant artifacts, is it possible to have the associated source tarballs include, recursively, the submodules in my repo? Right now this isn't the case (same with the tarballs GitHub provides as well), and so the program isn't able to be compiled from just the source tarball accompanying releases. However, I'd like the release tarballs to be "stand alone" and capable of being built directly.

Is such a thing possible? If so, how might I go about it?

Thanks! Rob

[mistydemeo]

I looked into this when we added source tarballs, and unfortunately it looks like the answer is it's possible but it's not simple. :/ We lean on git archive, git's builtin tarball generation tool, which is the same thing used by GitHub. git archive is completely submodule unaware, so we'd need to iterate over submodules and manually add their contents to the main tarball in order to create a true submodule-aware source tarball.

It's not on our roadmap at the moment, but I'll tag this as a feature request and we can look into it at some point in the future.

[rob-p]

Thanks for the quick response! It looks like there are some bash scripts / git archive wrappers that do this, but I am guessing the interaction between cargo-dist and git archive is at a lower or more fundamental level.

In the meantime, I wonder if there is a way to generate additional artifacts in a cargo-dist release that would e.g. run such a script. In this way, one could provide e.g. "source_all.tar.gz" artifacts that won't be made by the cargo-dist workflow itself, but perhaps generated by a workflow triggered by cargo-dist and automatically published by cargo-dist. Is this something that could be done with custom jobs at the build-global-artifacts-jobs or host-jobs steps? I'm not very familiar with custom jobs in cargo-dist so I'm not sure if that's possible.

[rob-p]

Ok, with a little bit of fidgeting, I've realized that making additional artifacts does, in fact, work, and is pretty easy if what you want is the recursive tarball. I was able to do the following:

• use the script I linked above as a resource in my repo.
• add a [[workspace.metadata.dist.extra-artifacts]] in my Cargo.toml and have it produce an artifact called source_recursive.tar.gz.
• add the appropriate command to build that with the script; in my case something like: build = ["bash", "scripts/git-archive-all.sh", "--format", "tar.gz", "--prefix", "reponame/", "source_recursive.tar.gz"]

This adds a source_recursive.tar.gz artifact to the releases page. It's certainly not anywhere near as clean as having the source tarball produced by cargo-dist include the modules recusively, but it's, perhaps, a quick fix for anyone else who may need this.

[mistydemeo]

Nice! That should definitely work. If you want it to be named source.tar.gz and replace the default, you could disable the builtin source tarball and then name your artifact source.tar.gz in Cargo.toml.

[rob-p]

Great idea! I’ve forked and updated the script I linked above to also compute the sha256 sum file. This works perfectly as a replacement for source.tar.gz and source.tar.gz.sha256!