search

clarity20 / tql

Terse Query Language
1 stars 0 forks source link

Input expression parser should be two-phase #1

Closed clarity20 closed 5 years ago

clarity20 commented 5 years ago

The precise determination of NCV-expression bounds should be moved to a preprocessing step, before we perform tokenization properly speaking. The lexical information we can derive from the NCV boundaries is very valuable and would allow us to unwind some of the messier parts of the parser code, such as backtracking through the output to make corrections to incorrect guesses.

clarity20 commented 5 years ago

The implementation will consist of a new function, delimitNCVsInString(string). It will return a new version of the string into which we will insert begin- and end-NCV markers. The markers will count off the NCVs just as they currently do, and the type of NCV will be identified by one of three token-type indicators that we will add to the "tt" enumeration in the tql_globals source file.

clarity20 commented 5 years ago

Commit id# 7d53154 is the implementation. The calling routine does not yet exercise this code, and the obsolete code needs to be cleaned up. This is covered in issue #2.