Design framework for user-entensible tokenization rules

shonfeder / tokenize

A tokenizer written in (SWI-)Prolog. It has some useful features and some flexibility and it might improve.

The Unlicense

11 stars 5 forks source link

Design framework for user-entensible tokenization rules #21

Closed shonfeder closed 5 years ago

shonfeder commented 5 years ago

Likely in the form of a multifile predicate, letting users define new DCG rules that will generate tokens.

Perhaps, in the future we could replace the use of an option list with a module of token rules for specifying what kind of lexing the user wants to carry out.

shonfeder commented 5 years ago

related #22

shonfeder commented 5 years ago

I think #24 should satisfy this need.