As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like:
var tokens []string
iter := tools.NewIterator(root, tools.PreOrder)
for n := range tools.Iterate(iter) {
token := uast.TokenOf(n)
if token != "" {
tokens = append(tokens, token)
}
}
and then by printing all the tokens get a "tokenized" version of particular file.
But uast.TokenOf returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.
As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like:
and then by printing all the tokens get a "tokenized" version of particular file.
But uast.TokenOf returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.