Dimension type expanded unit types

In response to opening up $type: dimension tokens units and allow more than px and rem, there are 4 paths forward:

Don’t expand the list. px and rem forever.
Expand the list one-by-one. Every unit added must be reviewed and approved, and standardized.
Keep the unit types restricted, but allow an escape hatch.
Allow any unit; remove all restrictions and validation.

@romainmenke made an excellent argument against 4. And instead proposed a path forward for 3: keep the current list of valid units restricted, but add an escape hatch to allow vendor-specific units where needed. The following describes that escape hatch:

We should have a fixed set of specified units and allow any other unit as long as it is vendor prefixed.

Then there can't be conflicts between tools as those would be different vendors. There also can't be conflicts between tools and future spec expansion.

It still leaves plenty of room for custom definitions.

Keep in mind that design token files are a carrier format. How a unit is encoded in a file doesn't dictate how it is exposed in a design tool or the final output for developers.

Figma can create a custom unit y that is exposed to designers as y and output to developers as y while encoding it as figma-y in token files.

If a custom, vendor prefixed, unit is popular and gains adoption among other tools it is also a clear signal that it is useful. Such a custom unit should then go through the spec process to standardize it.

Instead of figma-y as the encoded unit it could also be a separate prop.

{
  "value": 10,
  "unit": "y",
  "vendor": "figma"
}

Originally posted by @romainmenke in https://github.com/design-tokens/community-group/issues/244#issuecomment-2294724417

I'm not sold on the vendor property concept.

If the idea is that vendor will hold the name of whatever design tool produced the tokens file, that puts a pretty large burden on design token parsers/transformers to support and maintain mappings from each design tool's units to each target platform's units.

The only other use I can imagine for such a property would be for design tools to optionally denote that the unit is platform specific...

{
    "value": 5,
    "unit": "vh",
    "vendor": "web"
}

...but for this approach to be useful, design tools would all need to be consistent in their usage of both unit and the optional property. At that point we're back to needing to document units in the spec. While less important, the name of the property should probably also be reconsidered if this is the intended usage.

No the vendor property is only meant to be used as an escape hatch. The spec should cover all commonly needed units.

If a vendor is prototyping a new unit and wants to already start using this in token files, then they can create their own unit value and denote it with their name in the vendor property.

This allows tool creators to ship features without needing to wait for the standard to catch up. But also without causing naming conflicts with other tools or the spec itself.

We don't want multiple vendors coming up with multiple different definitions for x.

Ideally this never happens and every vendor always goes through the standards process. But this severely limits what can be tried in the field in a short time frame.

Even if a vendor starts prototyping while going through the standards process it is still prudent to use the vendor annotation. Maybe the unit name changes based on feedback from others in the community group.

Alternatively a vendor might have a unit that only makes sense in their ecosystem. They might still want to expose it through token files, but it wouldn't make sense to standardize.

I would strongly urge tools not to do this.

But if it does happen, then it is better to have an annotation as to who "owns" this non-standard unit.

This has nothing to do with the platforms that values will be used on.

A vendor might be figma or meta. But not web or android

@romainmenke thank you for the more in depth explanation.

No the vendor property is only meant to be used as an escape hatch. The spec should cover all commonly needed units.

How do you define a commonly needed unit? Are the spec authors open to standardizing platform specific units? My understanding was that there was a desire to keep the spec mostly platform agnostic.

Additionally, this explanation makes it sound like option 3 in @drwpow's original post is not so much its own distinct option, rather its an additional detail that could be paired with option 1 or 2. It might be helpful to update that list for clarity.

If a vendor is prototyping a new unit and wants to already start using this in token files, then they can create their own unit value and denote it with their name in the vendor property.

This allows tool creators to ship features without needing to wait for the standard to catch up. But also without causing naming conflicts with other tools or the spec itself.

Ideally this never happens and every vendor always goes through the standards process.

Alternatively a vendor might have a unit that only makes sense in their ecosystem. They might still want to expose it through token files, but it wouldn't make sense to standardize.

I would strongly urge tools not to do this.

I think you just outlined many of the reasons why browser's have moved away from using vendor prefixes for prototype implementations of new CSS features. "Nothing is more permanent than a temporary solution" as the saying goes. If the spec designate a new vendor property as the way to prototype new units, I think there's a good chance it ends up getting used a lot.

CSS spec implementers have solved this issue by shipping new implementations behind features flags in the end user environment (browsers). That approach works because CSS parsers are forgiving, they ignore lines they don't understand rather than throwing an error.

I think a similar approach would be a better way to address prototype/pre-spec implementations. The spec can allow parsers/transformers to ignore invalid tokens. If a vendor wants to add a unit that isn't in the spec yet, they just use it in the unit property as if it was valid and then end users can use a plugin or custom transform (perhaps written by the vendor) in the parser to handle it.

How do you define a commonly needed unit? Are the spec authors open to standardizing platform specific units? My understanding was that there was a desire to keep the spec mostly platform agnostic.

This is difficult to talk about in the abstract.

If you have a specific unit in mind, maybe better to open an issue for that?

I think that most units will have a common idea that exists in each platform. Something like vh isn't platform specific. vh as a CSS unit is platform specific, but the core idea, height of the viewport is something that makes sense in each context, even print.

Keep the unit types restricted, but allow an escape hatch.

Additionally, this explanation makes it sound like option 3 in @drwpow's original post is not so much its own distinct option, rather its an additional detail that could be paired with option 1 or 2. It might be helpful to update that list for clarity.

This only means that the list is well defined and that undefined units are invalid. It doesn't mean that the list must be kept small.

I think a similar approach would be a better way to address prototype/pre-spec implementations. The spec can allow parsers/transformers to ignore invalid tokens. If a vendor wants to add a unit that isn't in the spec yet, they just use it in the unit property as if it was valid and then end users can use a plugin or custom transform (perhaps written by the vendor) in the parser to handle it.

This is a nice idea but it does have very sharp edges.

browsers regularly ship buggy implementations
browsers regularly ship things too early

Each time they get it wrong it requires shipping use counters to analyse what the impact would be of breaking changes. Each time they have to cross their fingers and hope that developers haven't used a feature too eagerly.

The only browser that can effectively do this analysis is Chrome because of their oversized market share.

You can't have tools shipping support for a unit x and then standardize a differently defined unit x later without causing some breakage.

You have to measure how much you will break and have some allowed margin that is deemed acceptable.

A good example of this is : https://developer.chrome.com/blog/smooshgate

I am not saying that vendor prefixes are a good solution. And I am well aware of their history on the web :)

I think that token files are slightly different.

A major issue on the web is that most websites aren't actively maintained. They were created some time ago and are expected to keep working in future browser versions. Those who created the website aren't the same persons as those using it. Those experiencing issues with a broken websites are unable to fix those issues.

So a removal of a vendor prefixed CSS property is extremely risky. It will break websites and it will affect many people who are unable to resolve the resulting issues on their own.

This is not true for token files.

the token file schema can have versioning
token files can have versioning
tools can have versioning
we can assume that token file creators and token file consumers can easily reach each other

In other words removing a tool specific unit can be handled much more easily and with much less dramatic impact that a property removal on the web.

I am also not particularly in favor of a vendor field.

However I do think these things are important:

the list of allowed units must be defined, all other units are invalid
more units can be specified in the spec as the need arises
tools must not squat common names for future units

I think that a vendor field is an effective solution for token files. If a better solution exists, then obviously we should pick the better solution :)

This is a nice idea but it does have very sharp edges.
* browsers regularly ship buggy implementations

* browsers regularly ship things too early
Each time they get it wrong it requires shipping use counters to analyse what the impact would be of breaking changes. Each time they have to cross their fingers and hope that developers haven't used a feature too eagerly.

That's not the case with feature flags, which is how most new features are prototyped. Because enabling the feature flag requires an end user to change their browser's settings, developers have no way of guaranteeing they'll be supported in for any user (even those that are using whatever browser has implemented the prototype). Because of this, developers don't ship those features in production, with the exception of small features that only serve as progressive enhancements or those that can be polyfilled by the developer shipping additional code.

A major issue on the web is that most websites aren't actively maintained. They were created some time ago and are expected to keep working in future browser versions. Those who created the website aren't the same persons as those using it. Those experiencing issues with a broken websites are unable to fix those issues.

I agree that "unmaintained but still in use" will probably be a less common problem for token collections than it is for the web, but I wouldn't be surprised if it still happens. Open source projects where a maintainer becomes disinterested and corporate environments where resources get shuffled away from something that's "working fine" to something more pressing both spring to mind as ways this could happen.

the token file schema can have versioning

It could, but I'd caution going down that route. Versioning the spec (as opposed to it being a backwards compatible living document) could cause a lot of churn for users. Two ways this manifests:

Support gaps. Various tools will roll out support for new versions of the spec at a different pace. That'll be the case no matter what, but versioning usually implies breaking changes. Breaking changes would force tools to choose between supporting different versions simultaneously or passing the burden of cross tool/verison compatibility onto users.

I realize breaking changes are already happening with things like value -> $value and changing some string values to objects. But I'm hoping that's only necessary because the spec is still currently in draft format and that once it stabilizes it will be backwards compatible.

Confusion around the version numbers themselves. As you mentioned:

token files can have versioning tools can have versioning

The latter is definitely already the case, and while the spec doesn't currently have a canonical method for the former, you could easily just use the file name to version them. The more things with versions numbers that are coupled, the harder it gets to have conversations without causing confusion. I believe that's one of the reasons docker has moved away from versioning the compose file format.

I'm starting to feel like maybe I'm getting a little bit off in the weeds now though as this is probably tangential to the original issue. I'll try to reign it back in,

I am also not particularly in favor of a vendor field.

However I do think these things are important:
* the list of allowed units must be defined, all other units are invalid

* more units can be specified in the spec as the need arises

* tools must not squat common names for future units
I think that a vendor field is an effective solution for token files. If a better solution exists, then obviously we should pick the better solution :)

Totally agree with the mentality. I think all three of these goals can be achieved with option 2 from the original post:

Expand the list one-by-one. Every unit added must be reviewed and approved, and standardized.

I think the key with that approach will be spec authors being receptive to various platform needs in a timely and cooperative manner.

design-tokens / community-group

Dimension type expanded unit types #245