StackOverflowError in Scanner.fetchToken

[lefou]

I see this error message when I try to parse this file: https://github.com/com-lihaoyi/mill/blob/8903f225fabf6d4605a6a8d5f741910fcc0ed256/build.sc

Caused by: java.lang.StackOverflowError
         at com.virtuslab.using_directives.custom.Scanner.fetchToken(Scanner.java:539)
         at com.virtuslab.using_directives.custom.Scanner.fetchToken(Scanner.java:545)
         at com.virtuslab.using_directives.custom.Scanner.fetchToken(Scanner.java:545)
...
        at com.virtuslab.using_directives.custom.Scanner.fetchToken(Scanner.java:545)
        at com.virtuslab.using_directives.custom.Scanner.nextToken(Scanner.java:202)
        at com.virtuslab.using_directives.custom.Scanner.<init>(Scanner.java:32)
        at com.virtuslab.using_directives.custom.Parser.<init>(Parser.java:23)

This is the code:

val content = new String(os.read.bytes(buildSc), Charset.forName("UTF-8"))
val extractor = new SimpleCommentExtractor(content.toCharArray(), true)

val settings = new Settings(false, true)
val context = new Context(new ConsoleReporter(), settings)

val source = new Source(extractor.extractComments())
val parser = new Parser(source, context)
val parsed = parser.parse()
val usingDefs = parsed.getUsingDefs()

[tgodzik]

I see two problems:

• it seems we are not stopping after reaching the code section
• we are calling fetchToken recursively, which is a bad idea for a tokenizer

[KacperFKorban]

Hi, try using UsingDirectivesProcessor instead of SimpleCommentExtractor. SimpleCommentExtractor is more of a util class that takes comments that have already been preprocessed in some way. Try something like this:

val content = new String(os.read.bytes(buildSc), Charset.forName("UTF-8"))

val settings = new Settings(false, true)
val context = new Context(new ConsoleReporter(), settings)

val processor = new UsingDirectivesProcessor(context)
val usingDefs = processor.extract(content /* might need a .toArray here or sth similar */, true, false)

A good source of inspiration would be to see how scala-cli handles using directives. e.g. https://github.com/VirtusLab/scala-cli/blob/9b4f317c11daa30aeb2649cdf8b9496304b2a39f/modules/build/src/main/scala/scala/build/preprocessing/ExtractedDirectives.scala

This does not mean of course that this is not a bug. It seems that the Parser can't handle large amounts of whitespaces very well.

[lefou]

Hi, try using UsingDirectivesProcessor instead of SimpleCommentExtractor. SimpleCommentExtractor is more of a util class that takes comments that have already been preprocessed in some way. Try something like this:

val content = new String(os.read.bytes(buildSc), Charset.forName("UTF-8"))

val settings = new Settings(false, true)
val context = new Context(new ConsoleReporter(), settings)

val processor = new UsingDirectivesProcessor(context)
val usingDefs = processor.extract(content /* might need a .toArray here or sth similar */, true, false)

A good source of inspiration would be to see how scala-cli handles using directives. e.g. https://github.com/VirtusLab/scala-cli/blob/9b4f317c11daa30aeb2649cdf8b9496304b2a39f/modules/build/src/main/scala/scala/build/preprocessing/ExtractedDirectives.scala

This does not mean of course that this is not a bug. It seems that the Parser can't handle large amounts of whitespaces very well.

Thanks for the hint!

Isn't UsingDirectivesProcessor also using SimpleCommentExtractor in its implementation? Only difference is, that it is using the Visitor.

[lefou]

Thanks for fixing it. Can you please create a new release containing the fix?