In practice, most scan-rule actions are small combinations of a very few basic ideas:
put some of the match back (i.e. trailing-context)
update the scan condition via push, pop, or jump
yield a token in one of several ways:
with a particular semantic category, or
as itself (i.e. punctuation), or
capitalized, or
in consultation with a reserved-word table
as before, call some arbitrary function (possibly with a parameter)
Now suppose the internal structure of scanner rules were changed to allow sequences of generic actions, appropriately parameterized. Then in concept, more of the character of a language is expressible directly in terms of what you can put in the definition document. Features you don't use in a given rule "optimize to nothing". It facilitates support for embedding one language within another. It may also shrink the bootstrap code.
In practice, most scan-rule actions are small combinations of a very few basic ideas:
Now suppose the internal structure of scanner rules were changed to allow sequences of generic actions, appropriately parameterized. Then in concept, more of the character of a language is expressible directly in terms of what you can put in the definition document. Features you don't use in a given rule "optimize to nothing". It facilitates support for embedding one language within another. It may also shrink the bootstrap code.