Retain token stream so that checks can examine comments, etc.

Original issue created by eaftan@google.com on 2014-03-19 at 07:38 PM

There have been several proposed checks where we need to look at comments to make the correct decision about whether there is a bug. For example, the Google Style Guide says that a fallthrough in a switch statement must include a comment explaining that it is intentional. Another example is test cases where the intent is to catch a expected exception but there is no fail() in the case where the exception is not thrown.

One other use is for checks like LongLiteralLowerCaseSuffix in which we need to know what source the AST node actually corresponds to to determine if there is an error (i.e. the AST node only tells us this is a long literal, but not whether it ended with a lowercase L or uppercase L).

cushon / issues-import

Retain token stream so that checks can examine comments, etc. #241