[question]: Where do we debug custom parsers at? What point do they happen?

ontoneio commented 2 weeks ago

I am using v4 and I am curious as to how to go about debugging my custom parser? I have a separate module that I wrote the parser in and am registering it in an adaptor class like so.

When I got to use the buildAllPlatforms function I can see that I am in fact calling the correct API's with the custom regsitered parser, but at no point in the lifecycle of the build do I find where the custom parser is supposed to take effect. I feel like I am missing some magicial configuration incantation that is not documented that will get me where I need, but I am not sure and I am going a bit crazy wondering why this isn't working. I would just like to be able to find the location to inject my debugger and see how my parser is working to get a custom format of of a JSON file.

Any help is greatly appreciated.

Adaptor

// adaptor class
import StyleDictionary from 'style-dictionary';
import { registerTransforms } from '@tokens-studio/sd-transforms';
import { Config, PlatformConfig } from 'style-dictionary/types';
import { tokenParser } from './tokenParser';

export default class TokenAdaptor {
  private baseConfig: Config;
  private styleDictionary: StyleDictionary;

  constructor(baseConfig: Config) {
    this.baseConfig = baseConfig;
    this.styleDictionary = this.setupConfig();
    this.init();
  }
  // Initialized on 'new' constructor keyword
  private setupConfig(): StyleDictionary {
    let styleDictionary;
    registerTransforms(StyleDictionary);
    StyleDictionary.registerParser({ ..tokenParser });
    styleDictionary = new StyleDictionary(this.baseConfig, { init: false });
    return styleDictionary
  }

  private init() {
    this.styleDictionary.init();
  }

  public async buildAllPlatforms() {
    await this.styleDictionary.buildAllPlatforms();
  }
}

Parser

// custom parser
// implementation is trivial
import { Parser, DesignTokens } from 'style-dictionary/types';
import { resolveReferences } from 'style-dictionary/utils';

export const tokenParser: Parser = {
  name: 'custom-parser',
  pattern: /\.json$/,
  parser: (options): DesignTokens | Promise<DesignTokens> => {
    const { filePath, contents } = options;
    try {
      const filePrefix = filePath?.replace('.json', '');
      const jsonData = JSON.parse(contents);

      // Resolve internal references within the file
      // const resolvedData = resolveReferences(jsonData, []);

      // Convert the resolved data to design tokens
      const designTokens: DesignTokens = convertToDesignTokens(resolvedData, filePrefix);

      return designTokens;
    } catch (error) {
      console.error(error);
      throw error;
    }
  },
};

Tests

// Test
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import TokenAdaptor from './tokenAdaptor';
import { Config } from 'style-dictionary/types';
import { existsSync, rmdirSync } from 'fs';
import { join } from 'path';

describe('TokenAdaptor', () => {
  const baseConfig: Config = {
    // Define your base configuration here
    include: [
      '../tokens/base/base.json',
      '../tokens/base/color.json'
    ],
    source: [
      '../tokens/base/base.json',
      '../tokens/base/color.json'
    ],
    preprocessors: ['token-studio'],
    platforms: {
      css: {
        transformGroup: 'css',
        buildPath: './build/css/',
        files: [
          {
            destination: '_variables.css',
            format: 'css/variables'
          }
        ]
      }
    }
  };
  const additionalConfig: Config = {
    // Define additional configuration here
    include: ['../tokens/additonal/additional.json'],
    source: ['../tokens/additonal/additional.json'],
    preprocessors: ['token-studio'],
    platforms: {
      scss: {
        transformGroup: 'scss',
        buildPath: './build/scss/',
        files: [
          {
          destination: '_variables.scss',
          format: 'scss/variables'
          }
        ]
      }
    }
  };

  let adaptor:TokenAdaptor;

  beforeEach(() => {
    // Setup: Create a new instance of TokenAdaptor before each test case
    adaptor = newTokenAdaptor(baseConfig);
  });

  afterEach(() => {
    // Breakdown: Clean up any resources or reset any state after each test case
    // Example: Remove generated build output files
    const buildPath = join(__dirname, 'build');
    if (existsSync(buildPath)) {
      rmdirSync(buildPath, { recursive: true });
    }

  });

  it('should initialize with the base configuration', () => {
    expect(adaptor).toBeDefined();
  });

  it('should build all platforms', async () => {
    await adaptor.buildAllPlatforms();
    // Add assertions to check if the build output files were generated correctly
    const cssbuildPath = existsSync('./build/css/_variables.css')
    const scssbuildPath = existsSync('./build/scss/_variables.scss')
    expect(cssBuildPath).toBe(true);
    expect(scssBuildPath).toBe(true)
  });

  it('should build specific platforms', () => {
    await adaptor.buildPlatforms(['css']);
    // Add assertions to check if the build output files were generated correctly for the specified platforms
    const buildPath = existsSync('./build/css/_variables.css')
    expect(buildPath).toBe(true)
  });

});

jorenbroekema commented 2 weeks ago

Hey! We recently added some more information about when what happens in terms of the Style Dictionary lifecycle and all of its hooks: https://v4.styledictionary.com/info/architecture/ Hopefully the diagram helps clarify this.

To answer your question more directly, as soon as you call this.styleDictionary.init(); it will initialize the dictionary and do the following steps:

So after finding all of your token files by computing the glob array -> files array, for each token file, it'll run the registered parsers if their respective pattern props match the that file.

Two notes from reading your code:

resolveReferences utility is only meant to be ran on string values at the moment, it doesn't really work on a full dictionary object nor object-value tokens, it'll likely throw an error by the time it tries to do value.replace since .replace is a String method
convertToDesignTokens depending on what this method does, it might be interesting to run that method in a custom format and use the newly added formatPlatform to not build the output to the filesystem, but instead return it as an object datatype in your scenario. I'm happy to expand a bit on this approach once I know a little bit more on what your goal is to do with the design tokens

jorenbroekema commented 2 weeks ago

I'm going to create an issue for improving resolveReferences to handle object-value tokens. https://github.com/amzn/style-dictionary/issues/1238

We could consider adding a resolveReferencesDictionary utility that allows you to run it for the entire dictionary object, although I think this might not be necessary when you can accomplish this by just doing:

import StyleDictionary from 'style-dictionary';

StyleDictionary.registerFormat({
  name: 'js-object',
  format: ({ dictionary }) => {
    return dictionary.tokens;
  }
});

const sd = new StyleDictionary({
  source: [
    "tokens/**/*.json",
  ],
  platforms: {
    resolved: {
      files: [{
        format: 'js-object'
      }]
    }
  }
});

const platformFiles = await sd.formatPlatform('resolved');

// basically the original dictionary but with references resolved!
// also has "original.value" property if you want the unresolved value of a token
console.log(platformFiles[0].output);

amzn / style-dictionary