support sets of JSON files

[freemansoft]

This PR accepts a directory of design token json files and loads them into the same data structure used by the current single file code. The design tokens were generated using token zen garden https://tokenzengarden.design/explore/e70c7218-8978-4254-967f-4e119219ce5c

The PR adds more information to some of the throw exceptions. The current code eats some of the info required for troubleshooting when it re-throws.

PR for issue https://github.com/mark-nicepants/figma2flutter/issues/3

[freemansoft]

Added a check against the token "type" for "color". The color checker should have a list of color types probably. There is also a place where a string tries to guess if it is a color. :-(

[freemansoft]

This PR generates what I believe to be the correct tokens from a set of files as shown in the examples2 directory. The README in that directory documents the markup that didn't work before and still doesn't work. The sample files were manipulated to work within the constraints of the current parsers and approach.

[stongef-sonepar]

This is required for what I want to use as well. One issue i'm still facing, here's the exception :

dart bin/figma2flutter.dart --input ../Watts-Design-System/tokens/tokens.json --output lib/src/ui/
inputJson:  (global-tokens, spark-tokens, core, ..., $themes, $metadata)
setOrder: [global-tokens, spark-tokens, core, theme-sky/ui-element-color, theme-jungle/ui-element-color, - - - - - - - -/documentation]
themes: [Instance of 'TokenTheme', Instance of 'TokenTheme', Instance of 'TokenTheme']
Found 3 themes, generating code
Generating Theme: full
looked at hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.1100}) to see if it is a color - true
looked at 207 to see if it is a color - false
Originating exception stacktrace:
#0      _resolveColorValue (package:figma2flutter/models/token.dart:299:7)
#1      Token._resolveColorReferences (package:figma2flutter/models/token.dart:233:20)
#2      Token.resolveAllReferences (package:figma2flutter/models/token.dart:101:21)
#3      TokenTheme.resolve (package:figma2flutter/models/token_theme.dart:40:20)
#4      MappedIterator.moveNext (dart:_internal/iterable.dart:393:20)
#5      WhereIterator.moveNext (dart:_internal/iterable.dart:440:22)
#6      CastIterator.moveNext (dart:_internal/cast.dart:61:30)
#7      new _GrowableList._ofOther (dart:core-patch/growable_array.dart:202:26)
#8      new _GrowableList.of (dart:core-patch/growable_array.dart:152:26)
#9      new List.of (dart:core-patch/array_patch.dart:47:28)
#10     Iterable.toList (dart:core/iterable.dart:495:7)
#11     TokenTheme.resolvedTokens (package:figma2flutter/models/token_theme.dart:32:8)
#12     Processor.process (package:figma2flutter/processor.dart:22:30)
#13     _processTokens (file:///Users/fstonge/Development/Company/figma2flutter/bin/figma2flutter.dart:126:13)
#14     main (file:///Users/fstonge/Development/Company/figma2flutter/bin/figma2flutter.dart:88:20)
<asynchronous suspension>

ResolveTokenException{message: `base.color.black` defined as `{"value": hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.1100}), "type": "color", "path": ".base.color", "name": "black", "variableName": "baseColorBlack" }
` - Originating ResolveTokenException{message: Could not parse color for `207` originating from `base.color.neutral.hue`}}
Unhandled exception:
ResolveTokenException{message: `base.color.black` defined as `{"value": hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.1100}), "type": "color", "path": ".base.color", "name": "black", "variableName": "baseColorBlack" }
` - Originating ResolveTokenException{message: Could not parse color for `207` originating from `base.color.neutral.hue`}}
#0      TokenTheme.resolve (package:figma2flutter/models/token_theme.dart:43:7)
#1      MappedIterator.moveNext (dart:_internal/iterable.dart:393:20)
#2      WhereIterator.moveNext (dart:_internal/iterable.dart:440:22)
#3      CastIterator.moveNext (dart:_internal/cast.dart:61:30)
#4      new _GrowableList._ofOther (dart:core-patch/growable_array.dart:202:26)
#5      new _GrowableList.of (dart:core-patch/growable_array.dart:152:26)
#6      new List.of (dart:core-patch/array_patch.dart:47:28)
#7      Iterable.toList (dart:core/iterable.dart:495:7)
#8      TokenTheme.resolvedTokens (package:figma2flutter/models/token_theme.dart:32:8)
#9      Processor.process (package:figma2flutter/processor.dart:22:30)
#10     _processTokens (file:///Users/fstonge/Development/Company/figma2flutter/bin/figma2flutter.dart:126:13)
#11     main (file:///Users/fstonge/Development/Company/figma2flutter/bin/figma2flutter.dart:88:20)
<asynchronous suspension>

Here's an example of a color with a hue reference :

"base": {
      "color": {
        "black": {
          "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.1100})",
          "type": "color",
          "description": "base-colors-black"
        },
        "black-contrast-texticon": {
          "value": "{base.color.white}",
          "type": "color",
          "description": "base-colors-black-contrast-texticon"
        },
        "white": {
          "value": "hsla(0, 0%, 100%, 1)",
          "type": "color",
          "description": "base-colors-white"
        },
        "white-contrast-texticon": {
          "value": "{base.color.black}",
          "type": "color",
          "description": "--base-colors-white-contrast-texticon"
        },
        "neutral": {
          "100": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.100})",
            "type": "color",
            "description": "base-colors-neutral100"
          },
          "200": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.200})",
            "type": "color",
            "description": "base-colors-neutral200"
          },
          "300": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.300})",
            "type": "color",
            "description": "base-colors-neutral300"
          },
          "400": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.400})",
            "type": "color",
            "description": "base-colors-neutral400"
          },
          "500": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.500})",
            "type": "color",
            "description": "base-colors-neutral500"
          },
          "600": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.600})",
            "type": "color",
            "description": "base-colors-neutral600"
          },
          "700": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.700})",
            "type": "color",
            "description": "base-colors-neutral700"
          },
          "800": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.800})",
            "type": "color",
            "description": "base-colors-neutral800"
          },
          "900": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.900})",
            "type": "color",
            "description": "base-colors-neutral900"
          },
          "1000": {
            "value": "hsl({base.color.neutral.hue}, {saturation.100}, {luminosity.1000})",
            "type": "color",
            "description": "base-colors-neutral1000"
          },
          "100-contrast-text": {
            "value": "{base.color.black}",
            "type": "color",
            "description": "base-colors-neutral100-contrast-text"
          },
          "200-contrast-text": {
            "value": "{base.color.black}",
            "type": "color",
            "description": "base-colors-neutral200-contrast-text"
          },
          "300-contrast-text": {
            "value": "{base.color.black}",
            "type": "color",
            "description": "base-colors-neutral300-contrast-text"
          },
          "400-contrast-text": {
            "value": "{base.color.black}",
            "type": "color",
            "description": "base-colors-neutral400-contrast-text"
          },
          "500-contrast-text": {
            "value": "{base.color.black}",
            "type": "color",
            "description": "base-colors-neutral500-contrast-text"
          },
          "600-contrast-text": {
            "value": "{base.color.white}",
            "type": "color",
            "description": "base-colors-neutral600-contrast-text"
          },
          "700-contrast-text": {
            "value": "{base.color.white}",
            "type": "color",
            "description": "base-colors-neutral700-contrast-text"
          },
          "800-contrast-text": {
            "value": "{base.color.white}",
            "type": "color",
            "description": "base-colors-neutral800-contrast-text"
          },
          "900-contrast-text": {
            "value": "{base.color.white}",
            "type": "color",
            "description": "base-colors-neutral900-contrast-text"
          },
          "1000-contrast-text": {
            "value": "{base.color.white}",
            "type": "color",
            "description": "base-colors-neutral1000-contrast-text"
          },
          "hue": {
            "value": "207",
            "type": "color",
            "description": "base-colors-neutral-hue"
          }
        },
}

[freemansoft]

I'm pretty sure this is related to https://github.com/mark-nicepants/figma2flutter/issues/9 . The current parsing code supports only '+-*/' and no function calls. I ran into this with roundTo() on colors and had to pull them out. That probably needs to be replaced with a parser library

token.dart:_resolveMathExpression()

  Token _resolveMathExpression(Map<String, Token> tokenMap) {
    final isMultiply = valueAsString!.contains(' * ');
    final isDivide = valueAsString!.contains(' / ');
    final isAdd = valueAsString!.contains(' + ');
    final isSubtract = valueAsString!.contains(' - ');

    // Split on expression and parse the left and right side
    // We search for: a space, then a valid operator (*/+-), then a space
    final splitted = valueAsString!.split(RegExp(r'\s[*/+-]\s'));
...

[stongef-sonepar]

hsl and hsla seems to be implemented.

In my opinion, it's because this cannot be parsed as a color, because 207 is not a color. It's a variable used in other definitions :

"hue": {
            "value": "207",
            "type": "color",
            "description": "base-colors-neutral-hue"
          }

[freemansoft]

Integrated changes from https://github.com/mark-nicepants/figma2flutter/pull/7

[freemansoft]

Yup. Hmmm.

I made a change that anything of type: color is treated as a color to fix a problem where the number 16 was being treated like a color when it was really just a value.

Nope that isn't it. I just tested with your data and that wasn't the issue.

[stongef-sonepar]

Integrated changes from #7

I will abandon my pr then, good job :)

[stongef-sonepar]

Thanks for helping me! I will get your branch and try to debug my issue.

[freemansoft]

Not sure that hsl() is supported. Looking in token.dart: _resolveColorValue() There is code specifically there to handle rgba but none to handle hsl

color_value:maybeParse() operates against rgb and hsla but not hsl. Could that be the issue?

[stongef-sonepar]

-Not sure that hsl() is supported. Looking in token.dart: _resolveColorValue() There is code specifically there to handle rgba but none to handle hsl-

color_value:maybeParse() operates against rgb and hsla but not hsl. Could that be the issue?

It could be, but the exception seems to point out that 207 is not a color. but it could be it.

[freemansoft]

I'll quit shooting in the dark on this now. 😭

hsl() isn't supported so it will have to be added similar to hsla(). That isn't the core problem though, as you pointed out. It is the attempt to convert 207 into a color.

I added this fragment of your JSON (converting hsl to hsla) to global.json in example2 and it fails with the same issue. If I remove base.color.black leaving only the hsl in base.color.white then it works. So yeah it is yet another "I think this value is a color" problem.

  "base": {
    "color": {
      "black": {
        "value": "hsla({base.color.neutral.hue}, 0%, 100%, 1)",
        "type": "color",
        "description": "base-colors-black"
      },
      "white": {
        "value": "hsla(0, 0%, 100%, 1)",
        "type": "color",
        "description": "base-colors-white"
      },
      "neutral": {
        "hue": {
          "value": "207",
          "type": "color",
          "description": "base-colors-neutral-hue"
        }
      }
    }
  }

[stongef-sonepar]

I manage to make it work, by allowing null values (instead of throwing) when a color cannot be parsed. Then I only add the parsed color if it's not null in the resolved array. I will clean my code a bit and create a commit in my repo, and send it to you to review.

[stongef-sonepar]

So there are two remaining issues on my end.

I have three themes, full, default and jungle. Default and Jungle should override properties of full, but right now they extends ColorTokens instead of FullColorTokens.

The other one is that I have a node called "[DS documentation]", and it generates properties like this : TextStyle get [DSDocumentation]TitleBlackTitle1;

I will work on it later but this is really promising.

[freemansoft]

You are seeing

• FullColorTokens extends ColorTokens
• DefaultColorTokens extends ColorTokens
• JungleColorTokens extends ColorTokens

But expect to see

• FullColorTokens extends ColorTokens
• DefaultColorTokens extends FullColorTokens
• JungleColorTokens extends FullColorTokens

Does it matter if the properties are generated correctly? I'm not sure you need a Dart class inheritance hierarchy.

[freemansoft]

The other one is that I have a node called "[DS documentation]", and it generates properties like this : TextStyle get [DSDocumentation]TitleBlackTitle1;

Looks like another one of the special character related issue. You see the same thing if you put a space in the theme name. You end up with a class whose name has a space in it which is invalid and doesn't compile.

[stongef-sonepar]

You are seeing

• FullColorTokens extends ColorTokens
• DefaultColorTokens extends ColorTokens
• JungleColorTokens extends ColorTokens

But expect to see

• FullColorTokens extends ColorTokens
• DefaultColorTokens extends FullColorTokens
• JungleColorTokens extends FullColorTokens

Does it matter if the properties are generated correctly? I'm not sure you need a Dart class inheritance hierarchy.

So I don't know if it's standard or not, but our file seems to have a "full" theme, which overrides all properties of abstract class ColorTokens. Default and Jungle only overrides some of these properties.

[stongef-sonepar]

Commit to support HSL : https://github.com/stongef-sonepar/figma2flutter/commit/1a021705cfae9e16940e70e74e91b67c5a02f4f8

[stongef-sonepar]

Commit to "not crash" when a color value cannot be parsed (I'm still not sure if this approach is good, we may need to rethink this) : https://github.com/stongef-sonepar/figma2flutter/commit/43743e491508cb070b56768b075596b50ac8c021

[stongef-sonepar]

Commit for special characters in variable names : https://github.com/stongef-sonepar/figma2flutter/commit/42d8f75dd7f6b5c9e013d249783938e4882340fa

[freemansoft]

@mark-nicepants This pr is feature complete and ready for review

[freemansoft]

Please review again.

[freemansoft]

What can I do to get this approved?