SublimeText / PackageDev

Tools to ease the creation of snippets, syntax definitions, etc. for Sublime Text.
MIT License
436 stars 83 forks source link

[Feature Request] YAML Macro system #110

Closed Thom1729 closed 7 years ago

Thom1729 commented 7 years ago

Example 1: a JavaScript syntax definition.

%YAML 1.2
%TAG ! macro:myMacros:
---
contexts:

    main:
      - match: !word function
        scope: keyword
        push:
          - !meta meta.function.js
          - function-body
          - function-arguments
          - !identifier entity.name.function.js

myMacros.py:

def meta(name):
    return [
        { 'meta_scope': name, },
        { 'match': r'(?=\S)', 'pop': True },
    ]

def word(match):
    return r'(?:%s){{b_after}}' % match

def identifier(scope):
    return [
        {
            'match': r'{{identifier}}{{b_after}}',
            'scope': scope,
            'pop': True,
        }
        { 'match': r'(?=\S)', 'pop': True },
    ]

Output:

%YAML 1.2
---
contexts:

  main:
    - match: "function{{b_after}}"
      scope: keyword
      push:
        - - meta_scope: 'meta.function.js'
          - match: r'(?=\S)'
            pop: true
        - function-body
        - function-arguments
        - - match: '{{identifier}}{{b_after}}'
            scope: 'entity.name.function.js'
            pop: true,
          - match: '(?=\S)'
            pop: true

Example 2: a color scheme:

%YAML 1.2
%TAG ! macro:colorSchemeMacros:
---
!remove variables:
  text: '#D8DEE9'
  mint: !hsb [ 114, 26, 78 ]

settings:
- name: Comment
  scope: comment, punctuation.definition.comment
  settings:
    foreground: !darken [ 0.5, !var text ]
- name: String
  scope: string
  settings:
    foreground: !var mint

This system uses YAML tags as macros. You can define any macros you like in a python module and refer to them using the %TAG directive. This is standards-compliant YAML, and the tag namespacing should forestall any conflicts.

We would need a build system to compile files with macros. In addition, we may want to provide specialized use cases, such as compiling a color scheme YAML file and then converting it to a plist.

I have a working implementation of the macro compiler in about 60 lines of python. Right now, it takes in a filename ending in .source and saves the result with that extension stripped. This API could be improved, particularly to make filename conversion more flexible. Once the API is hashed out, this feature is pretty much ready to go.

Thom1729 commented 7 years ago

Appendix: the current implementation.

import yaml
from os import path
import sys
import imp

filename = sys.argv[1]

output_path, extension = path.splitext(path.basename(filename))

if extension != '.source': raise "Not a .source file!"

def load_macros(macro_path):
    search_path, name = path.split(path.abspath(macro_path))

    fileObject, file, description = imp.find_module( name, [ search_path ] )
    module = imp.load_module('macros', fileObject, file, description)

    return [
        (name.rstrip('_'), func)
        for name, func in module.__dict__.items()
        if callable(func) and not name.startswith('_')
    ]

def apply_transformation(loader, node, transform):
    try:
        if isinstance(node, yaml.ScalarNode):
            return transform(loader.construct_scalar(node))
        elif isinstance(node, yaml.SequenceNode):
            return transform(*loader.construct_sequence(node))
        elif isinstance(node, yaml.MappingNode):
            return transform(**loader.construct_mapping(node))
    except TypeError as e:
        raise TypeError('Failed to transform node: {}\n{}'.format(str(e), node))

def get_constructor(transform):
    return lambda loader, node: apply_transformation(loader, node, transform)

input_file = open(filename, 'r')
for token in yaml.scan(input_file):
    if isinstance(token, yaml.tokens.DocumentStartToken):
        break
    elif isinstance(token, yaml.tokens.DirectiveToken) and token.name == 'TAG':

        handle, prefix = token.value
        if not prefix.startswith('macro:'): break
        macro_path = prefix.split(':')[1]

        for name, transform in load_macros(macro_path):
            yaml.add_constructor(prefix+name, get_constructor(transform))

syntax = yaml.load(open(filename, 'r'))

output_file = open(output_path, 'w')
yaml.dump(syntax,
    stream=output_file,
    version=(1,2),
    tags=False,
)
FichteFoll commented 7 years ago

The idea isn't too bad. Ideally you would want the format to support most of your actions natively, but this design allows it to do almost anything, which can be useful in some situations still.

It won't make it into 3.0.0 though because I want to focus on feature parity first. Even then, I would be more inclined to develop this as an external tool.

For color schemes I recommend CSScheme, although this is more of a by-product.

Thom1729 commented 7 years ago

The macro functionality has been implemented separately and is available here:

https://github.com/Thom1729/YAML-Macros