Local Cargo registry index Git error on shared volume

[jimmycuadra]

Apologies for the complicated nature of this issue. The root cause is probably not really related to Cargo, but because it happened to me while trying to use Cargo, I thought it'd be useful to track here in case anyone ever encounters the same issue. With that said...

I'm working on a Rust program that requires an instance of etcd for integration testing. To make everything easy and portable, the test suite runs via two Docker containers orchestrated through Docker Compose (formerly Fig). The container running the tests uses my own Docker image for Rust which as of the time I'm writing has Rust 1.0.0-beta.3 installed and the containing host directory mounted at the path /source inside the container.

The down side of doing this is that every time the container exits, the .cargo directory is wiped out, so each invocation of the test suite must re-resolve and download all the dependent crates. To alleviate this problem, I'm attempting to use a directory on my host machine as the container's .cargo directory via a volume mount, so it can persist across multiple invocations of the test suite, each in a new container.

The problem is that when the .cargo directory inside the container is on a volume shared from the host, Cargo is aborted from a Git permissions error during the download process:

root@59c9b7f2b312:/source# cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2] Failed to increase size of pack file '/source/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_VzVv1F': Operation not permitted

Here are the permissions of that directory:

root@59c9b7f2b312:/source/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack# ls -la
total 0
drwxr-xr-x 1 1000 staff 102 May  5 07:00 .
drwxr-xr-x 1 1000 staff 136 May  5 07:00 ..
-r--r--r-- 1 1000 staff   0 May  5 07:00 pack_git2_VzVv1F

And the output of df -T, for reference:

root@59c9b7f2b312:~# df -T
Filesystem     Type   1K-blocks      Used Available Use% Mounted on
rootfs         rootfs  19049892   1927336  16131832  11% /
none           aufs    19049892   1927336  16131832  11% /
tmpfs          tmpfs    1028252         0   1028252   0% /dev
shm            tmpfs      65536         0     65536   0% /dev/shm
none           vboxsf 243924880 236556804   7368076  97% /source
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/resolv.conf
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/hostname
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/hosts
tmpfs          tmpfs    1028252         0   1028252   0% /proc/kcore

Note that I'm running Docker on OS X via boot2docker, meaning it's all happening inside a VirtualBox VM, hence the vboxsf filesystem for /source.

I've tried it both with a .cargo directory that's already been run/populated on the the OS X host, and without it existing prior to it being created by cargo build inside the container. I've also tried it using my host's default ~/.cargo mounted at /root/.cargo inside the container and placing the .cargo directory inside the project directory in both the host and container by overriding CARGO_HOME. All these approaches result in the same error.

The specific error message seems to come from libgit2 in indexer.c.

It seems like this issue is caused by some interaction between Git and VirtualBox (what appears to be the same issue is being discussed in libgit2/git2go#194).

Again, sorry for the complicated use case, but since Cargo uses a Git repository for registry downloads/cache, it's a problem the team might want to be aware of. Thank you!

[alexcrichton]

Wow, thanks for the thorough report! Is it possible to run a syscall tracing utility like strace or dtruss? That would help get a little more insight into precisely what error is being returned on which file, and perhaps enable making a smaller reproducible test case. It looks like the git2go issue also has a lot of good discussion though.

[jimmycuadra]

Here's the full output of strace: https://gist.github.com/jimmycuadra/7c840d4b989f686d8bca

The most likely relevant snippets:

umask(0)                                = 022
umask(022)                              = 0
open("/source/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_aQw3Jz", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
chmod("/source/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_aQw3Jz", 0444) = 0
stat("/source/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_aQw3Jz", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0

...

ftruncate(4, 11)                        = -1 EPERM (Operation not permitted)

[kaspar030]

I think I'm hitting a simliar issue trying to ship a rustup/cargo installation within a Docker container.

I want to pre-install rust at container build time, when the user that will actually use it is not known. So rustup etc. has to be run as root. The user will have a random uid (we pass it from the host using docker run -u $(id -u).

In order to give the user write permissions, I set umask to 002 and pre-create the cargo and multirust folders.

Now unfortunately the registry git repository checkout gets initialized without git's "core.sharedrepository" set, causing (I think) git to set the umask back to 022. For repos initialized with the git CLI, setting "core.sharedrepository" in the global git configuration has the desired effect (new repository is group read/writable), but the git repositories initialized by cargo don't seem to honour git's configuration.

edit This might be a missing feature of the libgit2 library used: https://github.com/libgit2/libgit2sharp/issues/1151 (wrong googling. the rust bindings seem to support creating of a shared repository by passing options the the repo initalization.)

[alexcrichton]

@kaspar030 oh so we could just clone repositories a little differently? I'd be totally down for tweaking that, do you have a specific option in mind?

[kaspar030]

Well, for the git cli git init --shared=group does the trick.

@alexcrichton I'm just realizing that you are the author of git2-rs, surely you know how best how to pass the shared option to libgit2?

Thing is, this has to be optional, maybe controlled through an env variable.

[kaspar030]

Well, for the git cli git init --shared=group does the trick.

I confused initializing and cloning, sorry.

For cloning, it is not as simple. Maybe initializing the repository as shared, then adding the clone source as remote would work.

[alexcrichton]

Ah sorry I'm not too familiar with --shared to git-init, and I'm not entirely sure what the issue here is as well. If the fix though is a piece of functionality in git2-rs or just exposing something already in libgit2, I'd be more than happy to add it to git2!

Cargo already controls all the checkouts of repositories and it's fine for it to configure them differently than git might otherwise normally do so (as Cargo controls anything), so tweaking any behavior here and there should be fine.