Token ID format is not ideal for consuming in hand written code

[romainmenke]

The current format for token id's, naming works great to keep them human readable. Which is generally preferred for GUI's.

But it is non-ideal for writing code that consumes design tokens.

{
    "group name": {
        "token name": {
            "$value": 1234
        }
    }
}

This example could be converted to CSS in this way :

Using a BEM like notation to make it valid CSS.

--group-name__token-name: 1234;

But this quickly falls apart :

{
    "group name": {
        "token name": {
            "$value": 1234
        },
                "token-name": {
            "$value": 1235
        }
    }
}

--group-name__token-name: 1234;
/* `-` character now needs to be escaped */
--group-name__token\\-name: 1235; /* escaping in css is so rare even the preview here falls apart */

Alternatively it could be a string :

order: design-token("group name.token name");

Both examples are problematic. Either significant burden is placed on code authors or on IDE's and tooling creators (possibly on both).

Finding a format that works for everything is near impossible. So maybe it needs to be embraced that this is going to be a problem?

The spec could formalise that token ID's should not be magically converted to a native variable/property format and only referenced by string.

This will make it easier to consume design tokens across multiple contexts without requiring code authors to memorise each conversion.

[TravisSpomer]

Converting names of things from one format to another is something devs do all day, every day. When naming variables and functions, we convert English words into a single strangelyCapitalizedIdentifier. React devs have to constantly switch back and forth between the css-name-for-a-property and the reactNameForTheSameProperty. When porting code from one language to another we rename everything so that it fits the new language better. And so on.

What's the proposed alternative, then? Just for a token named "font size", with no grouping, you'd typically use --font-size in CSS, fontSize as a variable name in C-like languages, FontSize in places that use PascalCase like WinUI, "font size" in Figma, and probably font_size in some convention I don't know about. Each of those users is going to be unhappy if the formatting of the token name doesn't match what they're used to, so it's clearly entirely impossible to pick one convention that would make even a majority of people happy without applying some kind of transform to the name.

It seems to me then that the benefit of such a rule is minimal, given that converting between naming schemes is a normal and very common thing we do constantly, and the downsides are very significant: either tokens are always consumed in a format that doesn't match language conventions (which is annoying), or they're accessed via freeform string keys (which is slow and error-prone, and not something most engineers would accept). And perhaps more importantly, it is absolutely guaranteed that lots of people would completely ignore the "rule" to never convert token names to another format: I would immediately break that rule, and I imagine most other engineers writing token tooling would too.

[romainmenke]

These are all examples of a human making a choice about converting English to something native to their context. I agree that this fine and something we do every day.

When we assign variable names we tend to have the full overview (or at least enough) to know that dropping space and - won't give conflicts. We might tweak the wording a bit so that it makes more sense in the current context.

But when a translation tool has to cover every possible character except .|{|} and convert it to something native to CSS, JS, ... it has to be lossy or escaped. Both are undesirable.

In the end designers might have to communicate back and forth with multiple teams until everyone is happy with the end result they get from their individual translation tools. This is a slow process.

Limiting the character set to something that always works for everyone is impossible. But dropping space, -, _, \, ?, (, ), [, ], ... would already help a lot.

[TravisSpomer]

But when a translation tool has to cover every possible character except .|{|} and convert it to something native to CSS, JS, ... it has to be lossy or escaped. Both are undesirable.

I guess I'm not understanding why you're saying those are undesirable. To me, lossy transformations of English concepts into identifiers in code is just very normal. And if your token name is so complicated that it's not immediately obvious how your preferred conversion tool is going to convert it into a valid identifier, and it becomes confusing, that seems like one of those "if it hurts when I do that, maybe I should just stop doing that" problems. It seems like a problem that just solves itself without needing additional rules.

(I do agree with you, by the way, that the current naming rules feel too lax! I don't really want to be writing code that can properly handle the inevitable case of someone putting emoji in the token name. My designer colleagues love putting emoji in names of things. In our design token system at work, the designers are insisting on lots of tokens with names that are just numbers. Those are very annoying to consume from certain types of code too, but the worst-case scenario is having to type something like Global.Color.Red["80"]. If things ever got worse than that, I would just tell them "sorry, no token names that are just numbers," or change the exporting process so that the code exports Red80 instead, or any number of other solutions. But those solutions don't require additional restrictions and complexity in the spec.)

[romainmenke]

With lossy I mean this for example.

In system one only a|b|- is allowed

In system two a|b|-|+|= is allowed.

With a conversion type that just drops characters that can't be converted, the value ab in system one can be one of these but an infinite amount of other options :

ab
a+b
+ab
ab+
a+=b
a=+b+==++==

---

You might decide to use - to indicate that a character was dropped.

• a+b -> a-b
• a=b -> a-b
• a-b -> a-b

Any lossy conversion is prone to collisions. And designers would need to know all the downstream translation tools and if they have a lossy conversion.

---

Escaping characters is possible in some contexts but very awkward to type.

---

Programs and contexts that don't involve a person typing references to tokens can use smarter ways around this.

Internally it can be a pointer/reference, a temporary unique id or a properly encoded string value.

[TravisSpomer]

Sure, I understand what lossy means. I just don't understand why you're seeing that as a negative. Any design token system is necessarily going to involve a process of back-and-forth iteration until naming schemes are agreed upon; this is just part of that process. A name collision is a build error that requires a dev to tell a designer "hey you did something stupid." But it's not really any different from "hey we spell it 'grey' in our design system, not 'gray'" or anything else that requires back-and-forth. Building or updating a design system involves a lot of those conversations over the course of months and years, and no rule we can add to this spec will really help avoid that without making something else worse.

Anyway, I don't think I have anything else to add here. I think that what you're proposing makes a lot more sense as a rule for a specific design system and not as a rule in the spec. But I do not wish to be accused of bringing holleration or hateration to this reposit'ry—I'll let someone else speak up.

[romainmenke]

But I do not wish to be accused of bringing holleration or hateration to this reposit'ry—I'll let someone else speak up.

This dialogue has been very informative for me!

Any design token system is necessarily going to involve a process of back-and-forth iteration until naming schemes are agreed upon; this is just part of that process. A name collision is a build error that requires a dev to tell a designer "hey you did something stupid." But it's not really any different from "hey we spell it 'grey' in our design system, not 'gray'" or anything else that requires back-and-forth.

This actually is the issue for me :) By having such a lax id/name format the specification places extra burden on designers, developers and tooling.

Not only does an internal convention has to be agreed upon but also a format has to be found that works for everyone and every tool. A person can no longer intervene on their end but has to request changes to the designer, who then has to check with everyone if the change is going to break other parts.

Imagine publishing a set of design tokens anyone can use and getting an issue that some tokens have collisions in tool X. It would be near impossible to resolve without breaking it for someone else.

I don't think this can be avoided completely, but maybe some limits to the name/id are acceptable?

[c1rrus]

Design tokens are platform-agnostic. As @TravisSpomer already pointed out, when a design token is exported as code it's useful to convert the name into a format that's familiar for the target language (e.g. camelCase, kebab-case, etc.). That's exactly what existing design token export tools like StyleDictionary have been doing for some time already.

@romainmenke You are right that this can be a lossy conversion and, depending on the logic of that conversion, can lead to either naming clashes (e.g. 3 separate tokens named token name, token-name and ToKeN nAmE might all get camelCased to the same tokenName) or undesirable names with escaped characters (like in your example).

It's worth noting that in other contexts, like displaying token names in a design tool's GUI or a styleguide website, the "raw" token names could just be displayed directly without needing any conversion.

(That being said, choosing token names that differ only in whitespace or capitalisation is almost always going to be a bad idea as it makes them harder to distinguish)

However, for our spec we have intentionally chosen to keep the rules fairly relaxed. Our assumption is that different teams will have different naming conventions and, considering the international audience, may well want to use non-Latin scripts too (or emojis 😛!). We don't want to restrict that unnecessarily.

In other words, our spec aims to standardise how design token data is represented but not how you name or organise your tokens.

Furthermore, we don't want to mandate how names should be converted when exporting to code, as that too may vary from team to team. It's already the case that this is configurable in tools like StyleDictionary. So a token called Token Name may be exported to SASS as $token-name for one team, but $dsname-token-name for another.

I don't think the spec's lack of restrictions for token names is a problem though as there are various ways export tools can work around it:

• They can report an error users when different tokens would get exported using as the same name in code (this is something StyleDictionary already does today)
• Alternatively, they can work around it by adding an auto-generated prefix or suffix to the name is code (e.g. tokenName & tokenName2) - though I'd hope the tool would warn the user when this happens

Teams can then decide what to do. I suspect most will decide to revise their token names so that they avoid the problem.

I can also imagine that people might make linting tools for token files (similar to ESLint for JS or Stylelint for CSS). Tools like that could help flag potentially problematic token names early.

Perhaps they could also be configured to enforce a team's chosen naming conventions. E.g. if @TravisSpomer's team doesn't want to allow emojis in token names they could perhaps configure their linter to flag any tokens that don't adhere to that rule.

[romainmenke]

I really appreciate the inclusiveness behind lax token naming and I am hoping that this can work out. But I am also trying to be realistic and this will cause issues unless properly addressed by the specification.

---

I can also imagine that people might make linting tools for token files (similar to ESLint for JS or Stlyelint for CSS). Tools like that could help flag potentially problematic token names early.

Perhaps they could also be configured to enforce a team's chosen naming conventions. E.g. if @TravisSpomer's team doesn't want to allow emojis in token names they could perhaps configure their linter to flag any tokens that don't adhere to that rule.

This seems highly problematic to me :)

Requiring a 4th tool to make design tokens actually compatible between two systems, whereas design tokens are intended to be the compatibility layer seems counter productive.

Problems with token names are common if you look through the issues on style dictionary. The fact that they needed to add a couple of features to work around this exposes this flaw, it does not fix it :)

This also introduces vendor lock in issues. A specific context using dependency tokens will have these dependencies :

• design tokens specification (this is great)
• exact implementation of the translation tool
• internal team convention

If the translation tool is no longer maintained it will not be possible to just find a drop in replacement.

If the team wants to use a different convention it might also not be possible without switching translation tools.

---

It's worth noting that in other contexts, like displaying token names in a design tool's GUI or a styleguide website, the "raw" token names could just be displayed directly without needing any conversion.

In my opinion this should be weighted fairly against usage in non GUI contexts. If a certain specification design choice makes sense for designers in a GUI but causes harm for developers it should be reconsidered.

My fear is that design tokens will remain a fringe feature that too often can not be used by developers.

---

At the moment I am strongly leaning towards color: design-token('color.button.light blue') for a PostCSS plugin. This embraces that the naming isn't CSS compatible and removes any chance of conflicts.

It has the added benefit that any part of token path can be searched for in a tokens file (or in a GUI design tool).

Making this easy to type can be done with editor extensions.

---

Maybe a non normative section can be added to the specification with best practices for translation tools?

[romainmenke]

We embraced the fact that token ID's can have any characters and are not compatible with CSS.

Our current implementation looks like this :

https://www.npmjs.com/package/@csstools/postcss-design-tokens

@import url('./imported.css');
@design-tokens url('./tokens/basic.json') format('style-dictionary3');

.foo {
    font-family: design-token('font.family.base');
    font-size: design-token('size.font.small');
    color: design-token('color.font.base');
}

Autocomplete helps with all the typing (https://marketplace.visualstudio.com/items?itemName=RomainMenke.csstools-design-tokens)

Being able to use the exact naming from a design tool in CSS works really nice.