Our scalameta plugin doesn't capture all the sources

[blorente]

When executing semanticdb, we can see that there are some managed sources that do get compiled (registered into sbt), but do not generate semanticdb files. Here is an example:

Sources

borja@xmi ~/work/scala/tmp/ScalaByExample $ sbt "show sources" | awk -F"\\\\(|, |\\\\)|\\\\* " ' /(\(\/|\*)/ {for (i=2; i<=NF; i++) print $i}'
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/ScalaExamples/palindromes.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/ScalaExamples/Newton.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/ScalaExamples/iterites.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/ScalaExamples/Auction.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/PubSub/EventStream.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/PubSub/HostCatcher.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/PubSub/LinkCheck.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/PubSub/Logger.scala
/home/borja/work/scala/tmp/ScalaByExample/src/main/scala/com/joshroppo/PubSub/Main.scala

borja@xmi ~/work/scala/tmp/ScalaByExample $ sbt
[info] Loading global plugins from /home/borja/.sbt/1.0/plugins
[info] Loading project definition from /home/borja/work/scala/tmp/ScalaByExample/project
[info] Loading settings from build.sbt ...
[info] Set current project to ScalaExamples (in build file:/home/borja/work/scala/tmp/ScalaByExample/)
[info] sbt server started at 127.0.0.1:4604

Compilation run

sbt:ScalaExamples> clean
[success] Total time: 0 s, completed Mar 29, 2018 11:16:08 PM
sbt:ScalaExamples> compile
[info] Updating {file:/home/borja/work/scala/tmp/ScalaByExample/}scalabyexample...
[info] Done updating.
[info] Compiling 9 Scala sources to /home/borja/work/scala/tmp/ScalaByExample/target/scala-2.10/classes ...
[info] Done compiling.
[success] Total time: 5 s, completed Mar 29, 2018 11:16:16 PM

Semanticdb run

sbt:ScalaExamples> clean
[success] Total time: 0 s, completed Mar 29, 2018 11:22:09 PM
sbt:ScalaExamples> semanticdb
[info] Set current project to ScalaExamples (in build file:/home/borja/work/scala/tmp/ScalaByExample/)
List()
[success] Total time: 0 s, completed Mar 29, 2018 11:22:15 PM
sbt:ScalaExamples> 

My current hypothesis is that this is due to our plugin capturing sbt projects, and thus repositories that do not define projects are left out.

Some things to note:

• unmanagedSources is where most of the code of the application lives
• managedSources is where the generated code lives, for example, from using scalapb
• I don't think that distinction has anything to do with it. In the example above, the compiled code is in unmanagedSources
• One can use the following awk magic to extract the paths: sbt "show sources" | awk -F"\\\\(|, |\\\\)|\\\\* " ' /(\(\/|\*)/ {for (i=2; i<=NF; i++) print $i}'

[blorente]

Another point to consider, that project (ScalaExamples) is written in Scala 2.10, but our pipeline did not raise an error.

[blorente]

Okay, so the problem is in these lines of the plugin:

def relevantProjects(state: State): Seq[(ProjectRef, String)] = {
    val extracted = Project.extract(state)
    for {
      // For every project registered
      p <- extracted.structure.allProjectRefs
      // Get the full version
      version <- scalaVersion.in(p).get(extracted.structure.data).toList
      // Drop the patch number
      partialVersion <- CrossVersion.partialVersion(version).toList
      // Coerce to the version we want
      fullVersion <- V.get(partialVersion).toList
    } yield p -> fullVersion
  }

val extracted = Project.extract(state) extracts all projects correctly, but then this line in the for-comprehension (fullVersion <- V.get(partialVersion).toList) returns None, which gets transformed to List(), and thus the project is not considered and it fails silently.

I still need to investigate if this is specific to SBT 1.0.X.