Build error due to only a portion of git dependency being fetched

[lukerob01]

I am trying to build an app that depends on a custom git dependency that holds the datatypes to be used for communication by this app and others in the system. The datatypes dependency has bindings for several different languages and so has the following layout:

datatypes.git
   | - rust/
   |   | - Cargo.toml
   |   | - src/
   |   |   | - {modules for creating datatypes in rust}
   |   | - build.rs
   | - schemas/
   |   | - my_datatype.capnp
   | - {other language bindings}

In this repo, build.rs contains references to ../schemas/my_datatype.capnp to help build the language bindings for this datatype. The folders must be laid out like this so that each language binding has access the the same schema files.

In my app that I am writing, I add datatypes as a git dependency datatypes = { git = "ssh://git@github.com:22/.../datatypes.git" }. When I build in a shell with a standard cargo build, everything works fine but when I build it with the quick-start-simple template, I get an error that ../schemas/my_datatype.capnp doesn't exist in working directory /nix/store/{hash}-cargo-git/datatypes-0.1.0. Looking into this directory, I can see that it has only copied files from the rust subfolder of datatypes.git, not the whole repository like cargo build does.

Solution

The solution to this problem is when copying src files from git dependencies into the nix store, the entire git repository should be copied and the build should be run from the subdirectory containing Cargo.toml instead of only copying the files from this subdirectory.

I can provide more concrete code examples of anything needed on request

[ipetkov]

Hi @lukerob01 thanks for the report! I'm curious, is your project able to be built successfully via cargo vendor (e.g. cargo vendor >> .cargo/config.toml && cargo build)?

---

We emulate cargo vendor's behavior when downloading git dependencies by extracting each crate found in the repo. This effectively divorces the crate from the rest of the git repo. What gets confusing though is that vanilla cargo doesn't actually do this with git dependencies, it simply checks them out on disk and it happens that you can have sources referring to files outside of their crate directory successfully build.

One possible workaround you could try is adding a symlink in datatypes.git/rust/schemas -> ../schemas (then change your sources to refer to ./schemas and not ../schemas); then during extraction the symlink should get deeply copied into the extracted crate

[lukerob01]

Hi @ipetkov, thanks for such a helpful reply.

My current setup does not work with cargo vendor exactly as you predicted. Also, putting a symlink in datatypes.git/rust/schemas -> ../schemas allows cargo vendor to deep copy all the schemas into its dependencies and fixes the issue as you suggested it would.

However, when trying this new setup with crane and nix build I get a similar error to before since the schemas symlink in the nix store is not deep copied but just points to ../schemas which obviously doesn't exist. It appears that there is a slight discrepancy with cargo vendor in that symlinks are not being copied in.

Please let me know if there is anything else I can try out to help.

[ipetkov]

Please let me know if there is anything else I can try out to help.

This info has been super useful, thanks! Should have a fix for this in #367 :smile:

[lukerob01]

Thanks for getting this all sorted so quickly, I'm glad I could help. Its a great project and as a new nix user, I really appreciate all the effort you've gone through with the documentation.