Lexer is not properly tokenizing strings

[GoogleCodeExporter]

What steps will reproduce the problem?

Have import parser parse a source file containing lines like:

  fmt.Printf(" <import path=")
  fmt.Printf(" ")

What is the expected output? What do you see instead?

  Should not create an import entry, but does:

  )fmt.Printf(

Original issue reported on code.google.com by emerson....@gmail.com on 17 Jun 2011 at 6:56

[GoogleCodeExporter]

This is probably due to ImportParser in the package 
com.googlecode.goclipse.go.lang.parser having disregard for quotes and 
apostrophe.

Original comment by weidong...@gmail.com on 22 Mar 2013 at 2:50

[GoogleCodeExporter]

Agreed, all of the state machine parsers have needed to be reworked or 
rewritten for awhile.  

Original comment by st...@kryas.com on 22 Mar 2013 at 3:33

[GoogleCodeExporter]

I have been looking through the lexer/parser code, and I think they are quite 
brittle. I am thinking of extending the IContentAssistProcessorExt interface to 
include the appropriate methods so that all the parser tree information can be 
retrieved from the gocode server instead of using the internal lexer/parser 
code. Since Go itself already has libraries for parsing Go code, I think we 
might just make use of it instead of reinventing the wheels. What do you think?

Original comment by weidong...@gmail.com on 23 Mar 2013 at 2:10

[GoogleCodeExporter]

I agree the parsers are brittle and incomplete; I say go for it.  When
these Java parsers were written, the go parsers weren't as far along as
they are now.  However, keep this in mind when you are testing: the Java
parsers are really just pattern matchers and they do not break the same way
on incomplete syntax trees.  So, we still can look things up when the code
is in a bad state.  I feel this is important for any IDE.  If ever given
the time I was going to rewrite using ragel.

Original comment by st...@kryas.com on 23 Mar 2013 at 8:43