Please support sbt (simple build tool) used in the Scala world

mkurz commented 2 months ago

In the list of ecosystems you support sbt is missing. So you miss all the Scala projects out there :wink: (like https://github.com/playframework/playframework/)

https://www.scala-sbt.org/

If you need help I may can help out or we could ping the sbt maintainer(s) if they are available and willing to help.

vladh commented 2 months ago

Hey @mkurz, thanks for suggesting this. I don't see why we couldn't add this. However, I don't use Scala, so I would be grateful if you could help with one or two tips.

I'm currently looking at playframework's built.sbt — could you give me some pointers on how to identify the git repo associated with each dependency by looking at that file?

mkurz commented 2 months ago

In sbt, there isn't really a central place which can be parsed easily from the outside with a script or tool to get all the dependencies of a project. With sbt, you define projects to which you can add dependencies by adding them to the libraryDependencies list of each such project. The easiest definition probably looks something like:

lazy val `projectA` = (project in file("project-a"))
   ...
  .settings(
    libraryDependencies ++= Seq(
      "javax.inject" % "javax.inject" % "1",
      "com.typesafe" % "config" % "1.4.3",
      "org.slf4j" % "slf4j-api" % "2.0.16",
      "org.apache.commons" % "commons-email" % "1.5",
    ),
    ...
  )

lazy val `projectB` = (project in file("project-b"))
  ...
  .settings(
    libraryDependencies ++= Seq(
      "com.google.inject" % "guice" % "6.0.0",
      "com.h2database" % "h2" % "2.2.224",
      "org.hibernate" % "hibernate-core" % "6.5.2.Final",
      "org.awaitility" % "awaitility" % "4.2.1" % Test,,
      "org.assertj" % "assertj-core" % "3.26.0" % Test,,
      "org.mockito" % "mockito-core" % "5.12.0" % Test,
    ),
    ...
  )

Now if you look at how the Play Framework is set up: In project/Dependencies.scala we define lists of dependencies which we then reference in build.sbt. So any scala code in the project folder will just be compiled and made availabe to build.sbt. This is quite a powerful concept which give you lots of flexibility on how to set up and define sbt projects.

There are also sbt-plugins which can be activated on projects, or they even activate themselves automatically if certain conditions apply. Such sbt plugins can also add dependencies to a projects' libraryDependencies - this is not obvious however and you will not even see which dependencies get added without looking at the source of the sbt-plugin or by calling the sbt command to list dependencies.

IMHO the safest way to list all dependencies of a sbt project is by using the sbt command, e.g. there is sbt dependencyTree which would output a tree like:

$ sbt dependencyTree
[info] welcome to sbt 1.10.1 (Eclipse Adoptium Java 17.0.12)
[info] loading global plugins from /home/mkurz/.sbt/1.0/plugins
[info] loading settings for project play2-git-build-build from buildinfo.sbt ...
[info] loading project definition from /home/mkurz/work/play/framework/play2-git/project/project
[info] loading settings for project play2-git-build from plugins.sbt ...
[info] loading project definition from /home/mkurz/work/play/framework/play2-git/project
[info] loading settings for project Play-Framework from build.sbt,common.sbt ...
[info] resolving key references (48244 settings) ...
...
[info] org.playframework:play-exceptions:3.1.0-SNAPSHOT
[info] org.playframework:play-bom_2.13:3.1.0-SNAPSHOT
[info] org.playframework:play-jdbc-api_2.13:3.1.0-SNAPSHOT [S]
[info]   +-javax.inject:javax.inject:1
[info]   
[info] org.playframework:play-routes-compiler_2.13:3.1.0-SNAPSHOT [S]
[info]   +-org.playframework.twirl:twirl-api_2.13:2.1.0-M1 [S]
[info]   | +-org.scala-lang.modules:scala-xml_2.13:2.3.0 [S]
[info]   | 
[info]   +-org.scala-lang.modules:scala-parser-combinators_2.13:1.1.2 [S]
[info]   
[info] org.playframework:sbt-routes-compiler_2.12:3.1.0-SNAPSHOT [S]
[info]   +-org.playframework.twirl:twirl-api_2.12:2.1.0-M1 [S]
[info]   | +-org.scala-lang.modules:scala-xml_2.12:2.3.0 [S]
[info]   | 
[info]   +-org.scala-lang.modules:scala-parser-combinators_2.12:1.1.2 [S]
[info]   
[info] org.playframework:play-build-link:3.1.0-SNAPSHOT
[info]   +-org.playframework:play-exceptions:3.1.0-SNAPSHOT
[info]   
[info] org.playframework:play-configuration_2.13:3.1.0-SNAPSHOT [S]
[info]   +-com.typesafe:config:1.4.3
[info]   +-org.playframework:play-exceptions:3.1.0-SNAPSHOT
[info]   +-org.slf4j:slf4j-api:2.0.13
[info]   
[info] org.playframework:play-streams_2.13:3.1.0-SNAPSHOT [S]
[info]   +-org.apache.pekko:pekko-stream_2.13:1.0.3 [S]
[info]   | +-com.typesafe:ssl-config-core_2.13:0.6.1 [S]
[info]   | | +-com.typesafe:config:1.4.2 (evicted by: 1.4.3)
[info]   | | +-com.typesafe:config:1.4.3
[info]   | | 
[info]   | +-org.apache.pekko:pekko-actor_2.13:1.0.3 [S]
[info]   | | +-com.typesafe:config:1.4.3
[info]   | | 
[info]   | +-org.apache.pekko:pekko-protobuf-v3_2.13:1.0.3
[info]   | +-org.reactivestreams:reactive-streams:1.0.4
[info]   | 
[info]   +-org.reactivestreams:reactive-streams:1.0.4
[info]   
[info] org.playframework:play-run-support:3.1.0-SNAPSHOT
[info]   +-org.playframework:play-build-link:3.1.0-SNAPSHOT
[info]   | +-org.playframework:play-exceptions:3.1.0-SNAPSHOT
[info]   | 
[info]   +-org.playframework:play-file-watch:3.0.0-M3
[info]     +-io.methvin:directory-watcher:0.18.0
[info]       +-net.java.dev.jna:jna:5.12.1
[info]       +-org.slf4j:slf4j-api:1.7.36
[info]       
[info] org.playframework:play-framework_2.13:3.1.0-SNAPSHOT [S]
[info]   +-com.fasterxml.jackson.core:jackson-annotations:2.14.3
[info]   +-com.fasterxml.jackson.core:jackson-core:2.14.3
[info]   +-com.fasterxml.jackson.core:jackson-databind:2.14.3
[info]   | +-com.fasterxml.jackson.core:jackson-annotations:2.14.3
[info]   | +-com.fasterxml.jackson.core:jackson-core:2.14.3
[info]   | 
...

So you you can see the build defines many projecs (play-exception, play-routes-compiler,..) with each having different dependencies.

There are other commands to pipe the result into a file instead of outputting it in on the console (sbt "dependencyTree/toFile /tmp/deps.txt -f").

mkurz commented 2 months ago

Pinging @eed3si9n and @adpi2 - what do you suggest is the best way to output the dependency tree of an sbt project so https://thanks.dev/ can parse and analyze it?

eed3si9n commented 2 months ago

I have no strong opinion, but there have been a few related efforts, mostly for license enforcement / security vulnerability detection.

If you just want direct dependencies POM file might suffice?
GitHub's Dependency submission API - https://github.com/scalacenter/sbt-dependency-submission
CycloneDX SBOM / https://github.com/sbt/sbt/discussions/7340 - https://github.com/siculo/sbt-bom
SPDX SBOM?
Mend.io - https://github.com/lightbend/sbt-whitesource
https://www.scala-sbt.org/sbt-dependency-graph (now part of sbt) supports different formats.

vladh commented 2 months ago

Thank you for all of your suggestions. I admit that I find the variety of options, none of which seem totally straightforward, confusing as someone who doesn't use Scala.

I'm also still left with the question of how to turn an identifier such as org.playframework.twirl:twirl-api_2.13:2.1.0-M1 into a git URL, so that second-level dependencies (including authorship information) may be fetched.

adpi2 commented 2 months ago

Thank you for all of your suggestions. I admit that I find the variety of options, none of which seem totally straightforward, confusing as someone who doesn't use Scala.

sbt and other Maven-based build tools don't use a "lock" file, because the dependency resolution is reproducible. In order to get the list of dependencies you need to invoke sbt. Historically we created different ways of outputing the dependencies depending on different use case: human readability, specialized format (SBOM, GitHub dependency snapshots etc).

Probably the default way to output all dependencies (direct and transitive) is to use https://www.scala-sbt.org/sbt-dependency-graph. There are different output format you can use:

sbt dependencyTree or sbt dependencyList: it writes the dependency tree or list of each sub-project in the console
sbt dependencyDot: it writes a dependencies.dot file in each <sub-project>/target folder
sbt dependencyBrowseHtml: it writes a tree.json file in each <sub-project>/target folder

Notice that sbt-dependency-graph only outputs the runtime dependencies (not the dev or test dependencies).

Also, if you only need the direct dependencies it might be easier to generate and read the POM file: sbt makePom.

I'm also still left with the question of how to turn an identifier such as org.playframework.twirl:twirl-api_2.13:2.1.0-M1 into a git URL, so that second-level dependencies (including authorship information) may be fetched.

You can use the identifier to download the POM file on Maven Central: https://repo1.maven.org/maven2/org/playframework/twirl/twirl-api_2.13/2.1.0-M1/twirl-api_2.13-2.1.0-M1.pom. Then you should find the git URL in the scm tag:

<scm>
<url>https://github.com/playframework/twirl</url>
<connection>scm:git:https://github.com/playframework/twirl.git</connection>
<developerConnection>scm:git:git@github.com:playframework/twirl.git</developerConnection>
</scm>

vladh commented 2 months ago

@adpi2 Thank you for this! I think this is enough info to start work on this. I'll try to find time to do so. :)

thnxdev / issues

Please support sbt (simple build tool) used in the Scala world #1