shonfeder / tokenize

A tokenizer written in (SWI-)Prolog. It has some useful features and some flexibility and it might improve.
The Unlicense
11 stars 5 forks source link

Use conventional option processing #39

Closed shonfeder closed 5 years ago

shonfeder commented 5 years ago

Closes #13

The record-based approach used here is endorsed in https://eu.swi-prolog.org/pldoc/man?section=option

The next step is to drive new options down into the core tokenization phase. This will let us pass options to switch on/off numbers and strings, and later to customize their behavior.