Changes to support ProjectRef objects breaks multi-module project

[casualjim]

Hi

With the latest changeset 278f1b64 intellij doesn't recognize my own code anymore. going back one commit fixes everything (apart from stuff added through a git dependency).

Our project is not a trivial intellij project. it are 8 different sbt projects and from those 8 there are 2 projects with between 5 and 12 sub modules (sbt sub projects).

[mpeltonen]

Thanks for the report! I apparently need to test with more projects...

[mpeltonen]

And if you can provide any sample project that breaks, that would be great.

[casualjim]

I'll try again later with a simpler project

[casualjim]

I think you can trigger the case if you have a project reference to a sub project in a sibling directory which has a git dependency.

so

./library ./backend ./rl ./logback-akka ./scala-inflector

the ./backend being the main project and the core project there is defined as

val dispatchLiftJson =
   uri("git://github.com/mojolly/dispatch-lift-json")

val bcHome = file((sys.env get "BACKCHAT_HOME") getOrElse sys.props("backchat.home"))

def libraryProject(name: String) = ProjectRef(bcHome / ".." / "library", "library-%s" format name)

lazy val core = Project ("backchat-core", file("common/core"), settings =
    commonSettings ++ PB.protobufSettings ++ VersionGenPlugin.allSettings ++ Seq (
      description := "The common utilities for the BackChat platform",
      versionGenPackage := "backchat",
      compile in Compile <<= (compile in Compile, versionGen)(_ dependsOn _),
      aggregate in assembly := false,
      libraryDependencies ++= coreDeps,
      ivyXML := coreXml  ) ) dependsOn(
        dispatchLiftJson,
        libraryProject("bql"),
        libraryProject("email"),
        libraryProject("twitter"),
        libraryProject("facebook"),
        libraryProject("github"),
        libraryProject("io"),
        libraryProject("text"),
        libraryProject("activitystreams"),
        libraryProject("crypt"),
        libraryProject("metrics"),
        libraryProject("testing") % "test->compile"
      )

And the plugins are defined as.

object PluginsBuild extends Build {
  lazy val root = Project("plugins", file(".")) dependsOn (backchatSbt, scalateGenerate) settings (scalacOptions += "-deprecation")
  lazy val backchatSbt = file("../../library/backchat-sbt") 
  lazy val scalateGenerate = ProjectRef(file("../../xsbt-scalate-generate"), "xsbt-scalate-generator")
}

In library there are another number of projects that are referenced as ProjectRefs

val logbackExt          = bcHome / ".." / "logback-akka"
val rl                  = bcHome  / ".." / "rl"
val scalaInflector      = bcHome  / ".." / "scala-inflector"

lazy val core = Project ("library-core", file("core"), settings = libraryProjectSettings ++ Seq (
    description := "The common utilities for the BackChat library",
    libraryDependencies ++= coreDeps,
    ivyXML := coreXml
  ) ) dependsOn(logbackExt, rl, scalaInflector)

One of the projects in the libary codebase also adds dispatchLiftJson

What I'm seeing is that none of the classes in core/src/main are included in the project but the ones from core/src/test are.

[casualjim]

It seems the projects that depend on the backchat-sbt plugin in the project/project/plugins.scala are the ones that are affected. But those are also the ones without project references

[mpeltonen]

Thanks for the added details! Pretty non-trivial setup indeed :)

"I think you can trigger the case if you have a project reference to a sub project in a sibling directory which has a git dependency." <- And the actual problem is that the sub project having git dependency does not get Idea metafiles generated anymore (or they are somehow broken)?

[mpeltonen]

How many idea projects you have here, e.g. where do you run sbt-idea? Do you have aggregate project?

It would help to have diffs of all .iml files between a working version and broken version.

[mpeltonen]

I have sometimes debugged broken versions like this:

$ sbt gen-idea
$ rsync -a --include '*/' --include '*.iml' --exclude '*' . /tmp/tmp
... change config to use another plugin version ...
$ sbt gen-idea
diff -r . /tmp/tmp/|grep -v "Only in"

[casualjim]

I'll try to do this this weekend.

I run this in the parent of alll the projects.

#!/bin/zsh

if [ "x$MOJOLLY_HOME" = "x" ]; then
  echo "Did you source the mojolly env file? We need BACKCHAT_HOME and MOJOLLY_HOME to be set."
  exit 1
fi

libs=( rl scala-inflector logback-akka hookup )
backch=( library/backchat-sbt library xsbt-scalate-generate backend backchat-jruby  )
all=( $libs $backch )

for d in $all
do
  cd $MOJOLLY_HOME/$d
  sbt gen-idea
done
cd $MOJOLLY_HOME
sbt gen-idea

cp $MOJOLLY_HOME/src/intellij/*.xml $MOJOLLY_HOME/.idea
cp */**/.idea/libraries/* .idea/libraries

We don't use an aggregate project because when we set this up they didn't work all that well but I suppose if things work fine with the plugin we can use an aggregate project. I think a part of the problem is also in how our project setup is hacked together.

I'll post my diff in the next comment.

[casualjim]

In total it are about 40 projects: http://cl.ly/1a3l1g160J230S17181a

[mpeltonen]

Ok, thanks!

I think I understand now, at least partly. You get the pieces for one idea project out of the two generated by gen-idea. This is definitely kind of 'at your own risk' approach :)

But I think we could/should get the aggregates to work so that you'd get the whole setup with single gen-idea.

[casualjim]

I'll first try with the aggregate project at the parent level that may sort everything out if I understand correctly

[mpeltonen]

It might do that. But I just noticed that there's still one bug I need to fix. Now it only includes classes of Compile -scope in the classpath of referencing project. I need to add Test scope too.

[mpeltonen]

And I think I also need to find a way to add the project reference as module-library only if it is not also aggregate, as otherwise it will end up as both module and module-library...

[casualjim]

it's a long diff https://gist.github.com/99273257a8ac2179efd5

the aggegate build looks like this:

project/project/plugins.scala

import sbt._
import Keys._

object PluginsBuild extends Build {
  lazy val root = Project("plugins", file(".")) dependsOn (backchatSbt)
  lazy val backchatSbt = file("../library/backchat-sbt")
}

project/build.scala

import sbt._
import Keys._
import io.backchat.sbt._
import GlobalSettings._
import Resolvers._
import Dependencies._
import LocalProjectDependencies._

object BackchatAllBuild extends Build {

  lazy val root = Project("backchat-all", file("."), settings = buildSettings ++ Seq(
    version := "UNVERSIONED",
  //  compile := null, 
  //  test := {}, 
    publish := {}, 
    publishLocal := {},
    ivyLoggingLevel := UpdateLogging.DownloadOnly)) aggregate(
    logbackExt,
    rl,
    scalaInflector,
    backchatWebSocket,
    libraryProject("core"),
    libraryProject("bql"),
    libraryProject("email"),
    libraryProject("mongodb"),
    libraryProject("scalatra"),
    libraryProject("twitter"),
    libraryProject("facebook"),
    libraryProject("github"),
    libraryProject("activitystreams"),
    libraryProject("io"),
    libraryProject("testing"),
    libraryProject("metrics"),
    libraryProject("crypt"),
    libraryProject("text"),
    backchatProject("core"),
    backchatProject("models"),
    backendProject("common"),
    backendProject("billing"),
    backendProject("indexer"),
    backendProject("stream"),
    backendProject("tracker"),
    backchatProject("web"),
    backchatProject("kernel"),
    backchatJRuby
  ) dependsOn (
      logbackExt,
      rl,
      scalaInflector,
      backchatWebSocket,
      libraryProject("core"),
      libraryProject("bql"),
      libraryProject("email"),
      libraryProject("mongodb"),
      libraryProject("scalatra"),
      libraryProject("twitter"),
      libraryProject("facebook"),
      libraryProject("github"),
      libraryProject("activitystreams"),
      libraryProject("io"),
      libraryProject("testing"),
      libraryProject("metrics"),
      libraryProject("crypt"),
      libraryProject("text"),
      backchatProject("core"),
      backchatProject("models"),
      backendProject("common"),
      backendProject("billing"),
      backendProject("indexer"),
      backendProject("stream"),
      backendProject("tracker"),
      backchatProject("web"),
      backchatProject("kernel"),
      backchatJRuby
    )

}