Represent all space-preserver tokens in the same manner

clarity20 / tql

Terse Query Language

1 stars 0 forks source link

Represent all space-preserver tokens in the same manner #3

Closed clarity20 closed 5 years ago

clarity20 commented 5 years ago

Currently, space-preserver tokens inside NCV expressions use the "SPACE" token type indicator whereas other spaces are not marked up at all. We should use one convention for all space tokens to simplify the SQL-building process. Let us use the former convention; it is preferable that everything in the parsed string belong to some token.

clarity20 commented 5 years ago

Checkin # bb977c8 fulfills this task. All spaces are marked up in the preprocessing phase, and the main state machine passes these tokens to the final output stream unchanged.