Presently the lex() function is very optimistic, in that it only returns a channel of tokens. If an error is encountered (e.g. the text is not UTF-8 compliant), it can only return nil and no other indication of an error. Should have it return both the channel and an 'error'.
Presently the lex() function is very optimistic, in that it only returns a channel of tokens. If an error is encountered (e.g. the text is not UTF-8 compliant), it can only return nil and no other indication of an error. Should have it return both the channel and an 'error'.