Tracking issue for namespaced-features

[djc]

Docs: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#namespaced-features Original issue: #1286 Implementation: #5300

Not sure what the process is for stabilizing experimental Cargo features, I figured a tracking issue might be useful? @alexcrichton any guide on what should be done to move this towards stabilization?

Unresolved issues

• [x] Figure out how to deal with breakage of Cargo versions older than 1.19. #9161

About tracking issues

Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

[alexcrichton]

Indeed a tracking issue is useful! I think though this still has a large piece to implement which is to propagate namespaced features to the registry index

[djc]

Yes, I figured as much. So do you have any ideas/guidance about how to go about that? Are there established ideas on how to evolve the registry index?

[alexcrichton]

The evolution here is just a new key in the registry which we've done a bunch of times before, so it's fine to do whenever once there's an agreed up on design. Most of the time this is hard to undo though so we want to be pretty sure of the design before it lands. Technically it'll involve modifying crates.io as well as the publishing code in Cargo to send more data to the registry

[djc]

Can you be a bit more specific about which parts of the cargo and/or crates.io code need changing?

[alexcrichton]

You may want to hop onto the crates.io channel on discord as they may know more, but it's all around krate.rs and git.rs IIRC

[djc]

@alexcrichton okay, so, I'm still not entirely clear on what needs to be done on the crates.io side here. Is it just propagating the package.namespaced-features value? The feature values can still be serialized as a string, so that's the least invasive change. Alternatively, we could add more structure to the feature values and use serde's enum handling to have a new key that's more explicit about what's what.

Either way, I'm not sure how the compatibility with old cargo is supposed to work, or who all needs to be on board with a supposed design.

[alexcrichton]

@djc there's a few things here I think that we'd need to consider:

• One aspect is that we need to consider what happens when older versions of Cargo are used. For example if I use Cargo from 1.20.0 with a crate that has a namespaced feature, what happens? Is an error generated? Does the package not exist? Does it silently work sometimes and not others? My preferred way to solve this would be to add support for this today but don't actually allow publishing crates just yet. That way after a few cycles and Cargo rides the trains older Cargo will be able to understand newer crates in the future.
• Next is the actual registry format itself. Right now the features in the registry index are just an array of strings, but presumably yeah we'll need to propagate the namespaced features boolean as well. New registry fields could be used as well.

Before we go much further though I think it'd be good to get weigh-in from @rust-lang/cargo because once we start changing the registry it's sort of the point of no return

[joshtriplett]

@alexcrichton So, this is exactly the kind of change that really motivated the idea of supporting Cargo version dependencies in a way that older versions of Cargo can handle.

We can't make old versions of Cargo unable to parse new versions of the index; that's a serious problem.

If we're going to do this, we need to version the index, such that older versions of Cargo don't see crates that use this feature at all.

[alexcrichton]

@joshtriplett indeed! To be clear though the main issue is that an older Cargo won't understand the newer index, but we get to control how it doesn't understand the new index as well (aka no features, a mapping of features, or something like that). Having a fully versioned index seems far off still so I'd be fine implementing this before that comes along.

Now having a versioned index is a good idea though and we can start at any time to add safeguards into Cargo to avoid looking at future revisions!

[alexreg]

Is this going to move towards stabilisation soon?

[djc]

Probably not -- I'm still motivated to write code to move this forward, but I don't know exactly what needs to be done.

[alexreg]

Oh, I thought it was already implemented, just needed stabilising.

[djc]

The Cargo part is implemented -- it apparently needs complementary stuff in crates.io.

[alexreg]

Aha.

What I still don't understand is how this issue relates to implicit features for dependencies, exactly. (I was linked here from that issue.)

[ExpHP]

What I still don't understand is how this issue relates to implicit features for dependencies, exactly. (I was linked here from that issue.)

I agree that more could be said here. I find it odd how none of the examples I can locate for this feature show how it can be used to overcome the limitations of optional dependencies (which were its primary motivation)!

See this page:

• Docs for namespaced-features.

And now let me try to put it in context:

---

Suppose you have this:

[dependencies]
rand = { version = "0.5.5", optional = true }

Without namespaced-features, an optional dep implicitly defines a feature that cannot be overridden:

[features]
rand = [] # implicitly generated, always

...and rand is added to the dependency tree whenever the feature rand is required.

With namespaced-features, an optional dep implicitly defines a feature that can be overridden:

[features]
rand = ["crate:rand"]  # implicitly generated, unless you write your own rand feature

...and rand is added to the dependency tree whenever the feature crate:rand is required.

---

If you look closely, the actual namespacing of the dependencies does not, in itself, have anything to do with the issue of implicit features. It is the other changes enabled by the namespaced-features feature (notably the ability to override the implicit feature) that make a difference.

The justification for tying these changes together was given in https://github.com/rust-lang/cargo/issues/1286. Er... maybe. Actually, I can't really find it explicitly spelled out there. But the proposal is strictly more powerful than if we were to simply allow the implicit feature to be overridden with no other changes (because this also allows you to have a rand feature that does not toggle the rand dependency. Why? Idunno.).

[alexreg]

@ExpHP Thanks, that explanation makes a lot more sense to me. Was there a reason @sfackler's original proposal wasn't chosen though? I kind of like the idea of doing something like

[features]
a = { features = ["b", "c/d"], dependencies = ["c"] }

or such.

[tikue]

Is it possible to publish a crate that uses this feature? I'm getting:

error: api errors: invalid upload request: invalid value: string "crate:serde", expected a valid feature name at line 1 column 1862

[alexcrichton]

@tikue I believe not currently. In addition to crates.io checks the registry index isn't ready for namespaced features

[tikue]

@alexcrichton thanks, it was easy enough to work around, just had to name a serde feature serde1.

[alexreg]

@alexcrichton Does an issue for that exist on the crates.io repo?

[alexcrichton]

@alexreg I think that's this issue, the tracking issue for this feature. The feature needs to be designed for crates.io first.

[alexreg]

@alexcrichton Hmm? That's what I mean, we need to implement namespaced features on crates.io. There should be a separate issue for that over on that repo, I think, no?

[djc]

FWIW, I'd like to move this forward, but I'm currently having a hard time figuring out exactly what needs to be done about the index and crates.io. If someone can provide me with more guidance on that, that would be great.

[alexcrichton]

@djc I think that https://github.com/rust-lang/crates.io/pull/1487 may be helpful in providing a rough template for how to make progress?

[Ekleog]

Is there any news about this? https://github.com/rust-lang/cargo/issues/4548 still hits hard, the serde use case being the typical one.

[djc]

No news as far as I'm aware of. I'd be happy to do the implementation work, but I've had a hard time figuring out what exactly needs to be implemented. This probably depends on someone from the crates.io team on figuring that out. @sgrif would you have some time to collaborate with me on this?

[sgrif]

Sure. Ping me in Discord on Monday

[ghedo]

@djc @sgrif did you by any chance figure out what needs to be done to get this stabilized in the end?

[djc]

Yup, see https://github.com/rust-lang/crates.io/pull/2277. @ehuss why is this blocked on the new resolver?

[ehuss]

why is this blocked on the new resolver?

I'm not sure what you mean. Maybe because it is in the Project board? I probably did that just to organize things, not necessarily blocked.

One concern is that if we are making backwards-incompatible changes to feature resolution, we may want to gate them all under a single flag. We'd like to avoid littering Cargo.toml with a bunch of flags that change behavior. However, I'm not certain if that will apply here, since the new feature resolver is global, but these changes are local to the package.

[djc]

It would be nice if we can use the edition upgrade to flip defaults on some of these flags.

[ehuss]

It would be nice if we can use the edition upgrade to flip defaults on some of these flags.

Yep, we will likely do that. However, that might be a long ways away.

[djc]

Anyway, I'm fine with merging this into another flag if the timing lines up!

[ehuss]

One thing that has confused me about namespaced-features is why it really needs a special syntax. A concern is that the following might be confusing (especially for people already familiar with cargo):

# …
namespaced-features = true

[dependencies]
foo = {version="1.0", optional=true}

[features]
myfeat = ["foo"]

This doesn't enable the "foo" dependency, which it does today. It seems like it is unnecessary to prefix with crate:.

As an alternative, why not allow features and optional dependencies to coexist? Like you can say:

[dependencies]
url = "2.1.1"
serde = {version="1", optional=true}

[features]
serde = ["serde", "serde/derive", "url/serde"]

And just allow that to work?

I realize there were other things that crate: can do, like "hide" optional dependencies. But if that's the only reason to introduce new syntax, I'm not sure that's really worth it. (We could introduce other ways to hide them.)

[smarnach]

This doesn't enable the "foo" dependency, which it does today.

My understanding is that this does enable the foo dependency, since the feature myfeat depends on the implicitly defined feature foo, which in turn depends on crate:foo.

[dependencies]
url = "2.1.1"
serde = {version="1", optional=true}

[features]
serde = ["serde", "serde/derive", "url/serde"]

And just allow that to work?

This would introduce ambiguity. If we define another feature

myfeat = ["serde"]

what will this do? Will it enable the optional dependency serde, or will it enable the feature serde?

[djc]

What @smarnach said is also my understanding of the feature. I do think we should discuss the design before I address the issues you've filed. For those following along:

8044 Cargo cannot read index with namespaced features

(I do think the index has to know about namespaced features, if the index metadata is used to discover which transitive dependencies are to be built.)

8046 namespaced-features panics on unknown feature name

(This looks like a simple oversight that should error with a "non-existent feature" error.)

8047 namespaced-features allows implicitly named features with overlapping optional deps

I think this is intentional. The base idea was that the dependency namespace and the feature namespace were to be cleanly separated. However, I left in one loophole to cut down on boilerplate, which is that optional dependencies for which no feature of the same name exists get an implicitly defined feature of that name. So:

namespaced-features = true

[features]
<empty>

[dependencies]
serde = { version = "1.0", optional = true }

is equivalent to

namespaced-features = true

[features]
serde = ["crate:serde"]

[dependencies]
serde = { version = "1.0", optional = true }

I'm open to having a discussion about the desirability of this, though!

[ehuss]

My understanding is that this does enable the foo dependency, since the feature myfeat depends on the implicitly defined feature foo, which in turn depends on crate:foo.

Running a test with the example above and --features myfeat does not enable the foo dependency. Is that not working as intended? I filed #8047 for a similar issue, where it seems confusing that you can have a feature with the same name as an optional dependency, but they are unrelated.

what will this do? Will it enable the optional dependency serde, or will it enable the feature serde?

Both (kinda like it does today).

Just a caveat, that's just a rough idea, I just want to brainstorm if there is a way to avoid adding whether or not something is namespaced into the index. It's OK if old cargo's ignore crates with crate: in a feature, but it's (probably) not OK if they improperly resolve. The intent is to avoid changing the semantics of existing features, and only extend them with new meanings.

for which no feature of the same name exists get an implicitly defined feature of that name

This doesn't seem to be working as I understand it should. Printing out the FeatureMap:

"myfeat": [Feature("serde")],
"serde": [Crate("serde")],

If I run with --features myfeat or --features=serde, doesn't enable "serde" the crate. Maybe the dependency resolver isn't understanding it correctly?

[djc]

Running a test with the example above and --features myfeat does not enable the foo dependency. Is that not working as intended? I filed #8047 for a similar issue, where it seems confusing that you can have a feature with the same name as an optional dependency, but they are unrelated.

I seem to remember we had a plan to issue a warning for explicitly defined features of the same name as an optional dependency that did not include the crate:<feature> in the depended-upon features to make this better.

Just a caveat, that's just a rough idea, I just want to brainstorm if there is a way to avoid adding whether or not something is namespaced into the index. It's OK if old cargo's ignore crates with crate: in a feature, but it's (probably) not OK if they improperly resolve. The intent is to avoid changing the semantics of existing features, and only extend them with new meanings.

The conflicts in that case seem at odds with the original desire to undo the merging of the dependency and feature namespaces, which I (still) think would be a worthwhile goal that eventually makes everything easier to understand.

If I run with --features myfeat or --features=serde, doesn't enable "serde" the crate. Maybe the dependency resolver isn't understanding it correctly?

Yes, that sounds like a bug then. I'm guessing the feature is being spliced in too late in the process for the resolver to do the right thing.

[ehuss]

seem at odds with the original desire to undo the merging of the dependency and feature namespaces, which I (still) think would be a worthwhile goal that eventually makes everything easier to understand.

If an optional dependency implicitly creates a feature of the same name, then they aren't conceptually separate namespaces (all optional dependencies are also features of the same name). And adding a warning if you are missing crate:serde when redefining serde also implies they are coupled. And I think un-silencable warnings should generally be avoided. If that is an error, then it would further cement that they are not separate namespaces.

I've been thinking about this for a while, and I think I've been mislead a little by the current implementation. If it is really intended that optional dependencies always create a name = ["crate:name"] entry, then I don't think we'll need special index support. If things just aren't working as expected, maybe we can get all the issues fixed and then do some more tests to see how it works?

Another question I had: The original issue mentioned changing foo/bar syntax to enable the bar feature only if the foo crate is enabled. That change didn't seem to happen, was there some reason for that? Or is that not working as intended?

If we can keep it backwards compatible (where old syntax is interpreted the same), then we wouldn't even need the namespaced-features flag in Cargo.toml.

[smarnach]

It looks like there isn't a full consens yet what exactly this feature should look like. After reading https://github.com/rust-lang/cargo/issues/1286 I think this should go through a new, proper RFC process, and we should thoroughly the other options as well, including @sfackler's original proposal. I'm not very familiar with the RFC process, but I'd like to ask the Cargo team to discuss this in your next meeting.

We discussed https://github.com/rust-lang/crates.io/pull/2277 in the crates.io team meeting. Since this kind of change may be hard to revert for crates.io, we'd first like to finish the discussion on what exactly we want here before implementing this in crates.io.

[ehuss]

At a minimum this feature does need to go through some more work before making any changes on crates.io. We may want to consider an RFC, but beforehand we should at least:

• Have a clear description of what it changes and how it works. Also, the motivation and other considerations that are normally in an RFC. It's a bit hard to see what problems it is solving, how it should be behaving compared to how it currently does, and how that relates to the original #1286.
• Fix the bugs/issues in the current implementation.

I'd like to see an approach that doesn't require adding new fields to the index. I think that is feasible, we just need to avoid changing the meaning of existing, valid features.

[djc]

I don't think it necessarily makes sense to fix bugs and issues in the current implementation before we have an agreement on how the design should look -- otherwise I might have to change some things in the implementation that will then have to be changed again after we reach design consensus. At the very least, it might make sense to also post this to internals to get more exposure if we manage to reach consensus within this issue. So, mini-RFC:

Motivation

Optional dependencies implicitly masquerading as features causes two issues today, as described in #1286:

1. You cannot create features that have the same name as the optional dependency needed to support that features (which might be needed to, for example, "forward" the feature to another dependency), leading to ugly feature names which don't comply with the API guideline C-FEATURE.
2. The implicitly created features create a backwards compatibility hazard because it is not obvious when features originating from optional dependencies are removed.

Guide-level explanation

A feature is defined as a = ["b", "c/d"]. In this document, let's call a a feature. Both b and c/d are "feature values"; b enables another feature or an optional depenency and c/d enables a feature in a dependency.

This document proposes to make the difference between feature values targeting another feature and feature values targeting a dependency explicit by requiring a crate: prefix for dependency feature values. Because this is backwards incompatible (optional dependencies can no longer be enabled without the prefix), some explicit flag should be potentially be used to opt in to the new behavior.

With this proposal, features that need to forward the feature can trivially do so:

foo = ["crate:foo", "a-derive/foo"]

By requiring explicit features to be defined in order to control dependencies, it will be obvious when a feature is being removed, thus decreasing the backwards compatibility hazard.

Reference-level explanation

I'll skip this for now, because the above seems to cover things to large extent. Please feel free to point out things that should be clarified here. One thing that continues to be unclear to me is the interactions between cargo and the index. As far as I understand the index currently does not use the features data that it gets from cargo, so it seems to me that the impact should be limited.

Drawbacks

• Fixing the backwards compatibility hazard is an increase in boilerplate for the simple case where you want a feature to represent a single optional dependency. This could be mitigated by implicitly creating the feature for optional dependencies that do not have an explicitly specified feature. This solves motivation (1) but not (2). This variant is currently implemented.

• Switching to namespaced features will likely require a flag. While the ecosystem transitions, dependency graphs including crates with both style of metadata will need to interoperate.

Rationale and alternatives

The current behavior of mixing features and optional dependencies is compact, but can be hard to completely understand for more complex scenarios. In particular, having to introduce alternative names for features to work around clashes between feature and dependency names causes ugly and harder to find feature names that do not comply with the C-FEATURE guideline. Making features and dependencies explicitly distinct namespaces seems to keep most of the same advantages although potentially at the cost of more boilerplate for simple scenarios.

As an alternative, the original issue #1286 outlined a different way of handling this, where feature values are always dependencies or dependency features. If a feature wants to enable a different feature, a separate TOML table is specified for that feature, with fields for specifying its feature and dependency requirements.

Prior art

Gentoo: I'm familiar with Gentoo Linux ebuilds USE flag feature, which bears some similarities to Cargo features. In Gentoo, dependencies are fully separate from USE flags. Dependency edges can specify USE flags, and dependency (edge) USE flags can depend on current ebuild USE flags.

Unresolved questions

• For dependency feature feature values, should the dependency be implicitly enabled if the dependency feature is enabled? Or should it be an error to specify a dependency feature feature value without also explicitly enabling the dependency itself?

Future possibilities

If we introduce the crate: prefix for features, I would be interested to think about other future extensions to the feature model. For example, not: could be introduced to allow specifying mutually exclusive features, which seems like a desirable goal.

[ehuss]

Thanks for writing that up!

To be explicit, here is my understanding of how namespaced-features would be different from today's stable Cargo:

• It is now OK to define a feature with the same name as an optional dependency.
• Feature values (in the [features] table) can have the crate: prefix which refers to an optional dependency.

• An optional dependency foo creates an implicit feature of the same name if that feature name is not defined:

  [features]
  foo = ["crate:foo"] # Implicitly created if `foo` feature is not defined.

If that is all that it entails, then I don't think there needs to be an opt-in flag. This can be turned on for everyone, since it looks to be backwards compatible. And, IIUC, older cargos will ignore entries in the index if they contain crate: features (needs verification).

I also want to check, in the Unresolved questions, is this referring to if foo/bar is somewhere in the [features], should it automatically enable the optional dependency "foo"? Can you come up with concrete use-cases of real-world crates where it would be desirable to not automatically enable the dependency?

As for the "hiding" situation ("backwards compatibility hazard"), more concrete examples would be useful here, too. Like is the following a good example?

I want to expose a networking = ["crate:openssl"] feature, and avoid exposing the openssl feature because because I want to be able to change to different crates in the future, and openssl is not exposed in the API.

For that case, I can think of a few alternate solutions. One would be to ride on public/private dependencies. If an optional dependency is also private, then it does not automatically create a public feature of the same name.

Another solution would be: if crate:openssl appears in any feature, then the default openssl feature is not created. I'm not sure if that's too magical. I think that's fairly easy to describe, but I'm not sure if it is discoverable or memorable.

Sidenote: And it may make cfg expressions a little less explicit, because instead of cfg(feature="openssl") it would be cfg(feature="networking")). An alternative would be a "private" feature, where the cfg(feature="openssl") is still enabled, but external users won't see the openssl feature.

[CAD97]

I also want to check, in the Unresolved questions, is this referring to if foo/bar is somewhere in the [features], should it automatically enable the optional dependency "foo"? Can you come up with concrete use-cases of real-world crates where it would be desirable to not automatically enable the dependency?

One simple example, though not concrete:

I optionally support serialization via serde and arbitrary precision via num-bigint. The support for arbitrary precision involves some new types which should be serializable.

The impls are gatable by #[cfg(all(feature = "serde", feature = "num-bigint"))]. However, if I want to turn on the feature num-bigint/serde only when both my features serde and num-bigint are enabled, this is not possible unless I can write serde = ["crate:serde", "num-bigint/serde"] without enabling num-bigint itself.

---

Here's an informal (sorry, don't have the bandwidth to make it more formal) counter-proposal that doesn't cause the backwards-compatibility concerns:

"Just" allow defining a feature serde when an implicit feature serde for the dependency already serde exists. This would "just" add a list of features to also enable when enabling the serde feature, and it would still enable the optional dependency.

This resolves motivation 1 the obvious way (concern: users can't X; solution: let them X), and motivation 2 is at least partially solved by hiding the implicit feature for private dependencies.

[pksunkara]

This can be turned on for everyone, since it looks to be backwards compatible

Crates.io index stores only the defined features and not the implicit features. And because of that, I agree that this looks to be backwards compatible.

What are the next steps for this?

[djc]

Probably fix the bugs identified so far.

[ehuss]

I have posted a PR at #8799 to propose the new behavior which I described above which should make it backwards compatible (avoiding the opt-in) and fix all the known issues.

[est31]

Regarding the crate:foo syntax, I think it'll confuse a lot of people that crate:foo in Cargo.toml refers to the foo external crate (in fact, package), while crate::foo in mod.rs refers to the crate's internal module named foo.

Maybe it would be better to use pkg:foo instead of crate:foo?

[ehuss]

As discovered in #8832, this unfortunately breaks Cargo versions older than 1.19, even if they have a Cargo.lock file. We'll need to figure out how to address that concern before this can move forward.

[ehuss]

I forgot to post a followup here. #9161 implemented changes to support namespaced features in the index and avoid breaking older versions of cargo. There is some risk that versions older than 1.51 may resolve packages using namespaced features in such a way that those features don't get activated. I'll feel a little more comfortable once more time has passed so that 1.51 is at least a few months old.