cargo should not allow publishing incomplete sources

[ignatenkobrain]

The libc crate doesn't contain some parts of workspace directories and those are not present on crates.io (libc-test)... I think cargo should disallow such uploads, because it breaks cargo metadata.

[brain@ignatenko-w541 libc-0.2.20]$ cargo metadata --no-deps --manifest-path Cargo.toml
error: failed to read `/home/brain/Projects/upstream/rust2rpm/libc-0.2.20/libc-test/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

[alexcrichton]

Thanks for the report! This is currently intentional, however, as crate files don't contain sub-crates (for size reasons). In general the crate files are not intended to be unpacked and built themselves, but rather they need to be interpreted by Cargo.

[cuviper]

@alexcrichton This approach does make it tricky for distro packaging though, because if we use crates.io downloads as our source packages, then we can't build or test them. We only get to hope things still work when we compile leaf binaries which uses these dependencies.

Clearly Cargo knows how to compile these unpacked crate files. It would be really nice if we could present a local source dir and get Cargo to build/test it as if it were a crates.io package. e.g. ignore path-deps, workspaces, unselected optional deps...

[alexcrichton]

@cuviper oh I thought Cargo was always used as the middle-man here? Technically the format of a .crate file isn't even specified and can change over time, so assuming a tarball may cause breakage in the future.

My assumption is that for distros a scheme like cargo vendor is always used, in which case this is always a non-issue

[cuviper]

If there were a way to point Cargo at a raw foo.crate file for build/test/metadata, I'd almost be happy with that. But we do want the ability to patch sources too, so we probably still need to keep up with some ability to unpack these, however unspecified it may be. Perhaps then if we can repack our modified foo.crate and still point Cargo at it, that would be OK.

The package format seems a bit orthogonal though. Cargo is able to give special treatment to unpacked crates in directory registries too. So being able to treat any local path as such a crate would be nice.

My assumption is that for distros a scheme like cargo vendor is always used

Preferably not. We do allow some bundling nowadays, but still would rather have dependencies tracked on their own. e.g. if everyone vendors libgit2-sys, then everyone has to duplicate how to get it to use the system's libgit2.so. (A simple env, but that knowledge wouldn't be centralized.) Not to mention the usual arguments about tracking security updates and such.

[cuviper]

We're still forming Fedora policy, but see for example what Debian is trying to do: https://internals.rust-lang.org/t/debian-rust-packaging-policy-draft/4453

[alexcrichton]

Er well this probably isn't the location to re-litigate all packaging related concerns, but the goal here is to allow Cargo the flexibility to make changes in the future. As stated this report is a feature, not a bug in Cargo. It'd be pretty bad if when publishing the Servo script crate, for example, it included all other crates in the Servo repository!

The Debian packaging policy you've linked to is what I expected, which is a directory source. That directory source is assembled from cargo vendor which uses Cargo's own source code. Furthermore it doesn't run into this issue report at all.

If we want to go into a discussion of how to build .so libraries and build one crate from crates.io as a .so I think we're far enough away that trying to solve this report in that context is fruitless in the long run.

[cuviper]

I do agree with the broad point on this issue, that it should not package up the entire workspace. I'm trying to find a better middle ground where we can do what we need with crate packages standing alone.

We can take this discussion wherever you want, but this is incorrect:

The Debian packaging policy you've linked to is what I expected, which is a directory source. That directory source is assembled from cargo vendor which uses Cargo's own source code.

They're using a directory source, yes, but it's constructed from debian packages created with debcargo.

Furthermore it doesn't run into this issue report at all.

Because they're not trying to use cargo metadata, and they don't build/test library crates, only leaf binaries packages with cargo install. Personally, I don't even want to ship anything from a library build/test, there being no ABI and all, but I just want to answer "Does it work?"

[alexcrichton]

Sure yeah I'm definitely open to changes to Cargo! Also good point about using debcargo, I forgot about that!

If we want to talk about what Cargo needs to do, I've always found it far more useful to just start from basics. What is it you're trying to use Cargo for, and I guess specifically different from Debian? IIRC the Debian system meshed well with what Cargo supports today but did indeed have a drawback of only shipping binaries.