[Feature Request] YAML Macro system

[Thom1729]

Example 1: a JavaScript syntax definition.

%YAML 1.2
%TAG ! macro:myMacros:
---
contexts:

    main:
      - match: !word function
        scope: keyword
        push:
          - !meta meta.function.js
          - function-body
          - function-arguments
          - !identifier entity.name.function.js

myMacros.py:

def meta(name):
    return [
        { 'meta_scope': name, },
        { 'match': r'(?=\S)', 'pop': True },
    ]

def word(match):
    return r'(?:%s){{b_after}}' % match

def identifier(scope):
    return [
        {
            'match': r'{{identifier}}{{b_after}}',
            'scope': scope,
            'pop': True,
        }
        { 'match': r'(?=\S)', 'pop': True },
    ]

Output:

%YAML 1.2
---
contexts:

  main:
    - match: "function{{b_after}}"
      scope: keyword
      push:
        - - meta_scope: 'meta.function.js'
          - match: r'(?=\S)'
            pop: true
        - function-body
        - function-arguments
        - - match: '{{identifier}}{{b_after}}'
            scope: 'entity.name.function.js'
            pop: true,
          - match: '(?=\S)'
            pop: true

Example 2: a color scheme:

%YAML 1.2
%TAG ! macro:colorSchemeMacros:
---
!remove variables:
  text: '#D8DEE9'
  mint: !hsb [ 114, 26, 78 ]

settings:
- name: Comment
  scope: comment, punctuation.definition.comment
  settings:
    foreground: !darken [ 0.5, !var text ]
- name: String
  scope: string
  settings:
    foreground: !var mint

---

This system uses YAML tags as macros. You can define any macros you like in a python module and refer to them using the %TAG directive. This is standards-compliant YAML, and the tag namespacing should forestall any conflicts.

We would need a build system to compile files with macros. In addition, we may want to provide specialized use cases, such as compiling a color scheme YAML file and then converting it to a plist.

I have a working implementation of the macro compiler in about 60 lines of python. Right now, it takes in a filename ending in .source and saves the result with that extension stripped. This API could be improved, particularly to make filename conversion more flexible. Once the API is hashed out, this feature is pretty much ready to go.

[Thom1729]

Appendix: the current implementation.

import yaml
from os import path
import sys
import imp

filename = sys.argv[1]

output_path, extension = path.splitext(path.basename(filename))

if extension != '.source': raise "Not a .source file!"

def load_macros(macro_path):
    search_path, name = path.split(path.abspath(macro_path))

    fileObject, file, description = imp.find_module( name, [ search_path ] )
    module = imp.load_module('macros', fileObject, file, description)

    return [
        (name.rstrip('_'), func)
        for name, func in module.__dict__.items()
        if callable(func) and not name.startswith('_')
    ]

def apply_transformation(loader, node, transform):
    try:
        if isinstance(node, yaml.ScalarNode):
            return transform(loader.construct_scalar(node))
        elif isinstance(node, yaml.SequenceNode):
            return transform(*loader.construct_sequence(node))
        elif isinstance(node, yaml.MappingNode):
            return transform(**loader.construct_mapping(node))
    except TypeError as e:
        raise TypeError('Failed to transform node: {}\n{}'.format(str(e), node))

def get_constructor(transform):
    return lambda loader, node: apply_transformation(loader, node, transform)

input_file = open(filename, 'r')
for token in yaml.scan(input_file):
    if isinstance(token, yaml.tokens.DocumentStartToken):
        break
    elif isinstance(token, yaml.tokens.DirectiveToken) and token.name == 'TAG':

        handle, prefix = token.value
        if not prefix.startswith('macro:'): break
        macro_path = prefix.split(':')[1]

        for name, transform in load_macros(macro_path):
            yaml.add_constructor(prefix+name, get_constructor(transform))

syntax = yaml.load(open(filename, 'r'))

output_file = open(output_path, 'w')
yaml.dump(syntax,
    stream=output_file,
    version=(1,2),
    tags=False,
)

[FichteFoll]

The idea isn't too bad. Ideally you would want the format to support most of your actions natively, but this design allows it to do almost anything, which can be useful in some situations still.

It won't make it into 3.0.0 though because I want to focus on feature parity first. Even then, I would be more inclined to develop this as an external tool.

For color schemes I recommend CSScheme, although this is more of a by-product.

[Thom1729]

The macro functionality has been implemented separately and is available here:

https://github.com/Thom1729/YAML-Macros