sbt task system integration

[alexarchambault]

We now have a build module, that can be used as a library to build projects. Don't know how do-able / usable it would be to define a set of keys / tasks using our library to build scala projects. This keys / tasks set would probably be rather different than the current one in sbt, which would be incompatible with most of its plugins.

[sjrd]

sbt can already build Scala projects, has its own structure, ways to declare library dependencies, Scala versions, and everything. What is the expected benefit for users?

[alexarchambault]

As of today, breaking today's sbt plugins, the benefit would be limited. That would make sbt use bloop out-of-the box, and could provide a better transition from scala-managed projects to sbt ones.

It might be an interesting option if the sbt keys and internal model for building Scala were to evolve in the future though. (I was told about this could be a possibility for sbt 2.0.)

[Baccata]

Hi @alexarchambault , @sjrd , and everyone involved in this project.

I'm a developer who maintain (or help maintain) libraries that heavily cross build against several platforms and scala versions (including expecty). I've also contributed to ammonite, mill (including the bloop integration in mill), wrote a vscode extension to better the UX of looking for dependencies. I'm saying this to give some legiticimacy to this post: despite not being the most active contributor, I've tried to improve the state of the tooling at my own scale, with the little time I have available. I am not just coming here expecting for features to be worked on just because "it'd be nice/cool", and I try to give an opinion that respects the "big picture" of this project, as well as the desire to avoid scope creep.

I want to comment specifically on the two issues: #17 and #18. I have thought about commenting on #31, but it felt like this post wasn't completely on point with a discussion about "scope outlining".

The scala-cli project is really exciting to me, for a great number of reasons. One of them is how amazing the editor support is, even if scala-cli is not even officially out. Pulling dependencies is extremely gratifying because how fast the editor (vscode specifically, in my case) becomes aware of them. Not having to fiddle with a sbt or sc file, wait for it to recompile and load their respective models, only to have to wait for them to dump bloop configuration, is a breeze of fresh air. scala-clialready feels much more solid than ammonite in terms of integration with metals, and leaves me actually dreading having to open complex projects that are built with sbt or mill, where a change in the build can take (much) more than a minute before it's reflected in the editor. It also is greatly annoying to get the editor prompt for build reload after a change to the build file that does not impact anything that the editor should worry about.

As a user, I desperately want this UX for all my scala code, whether I edit a script, or a complex / cross-compiled project with 100+ targets.

I'm in agreement that scala-cli should remain extremely focused. I certainly wouldn't want for it to become yet another build tool. However, I believe it'd be quite interesting if build tools could integrate with it some way or another, to benefit from the slickness it offers in the editor / command-line (it could be, for instance, that build tools derive build settings (or even build structure from the declarative scala-cli configuration.)

Therefore I totally see (and agree with) the value that #17 and #18 would bring. I don't necessarily think that these integrations should be the responsibility of this project though, but I do think that scala-cli should empower build-tools to piggyback on it, thus offering a very smooth progression from "single file" to "complex project".

I don't think scala-cli would need to provide much more than it already does, partly because of what may (or may not) be an accidental decision : the use of the hocon format for configuration, which enables a number of interesting use cases.

For instance, as of today, it is possible to cross-compile "projects" using scala-cli, by leveraging hocon's include clauses :

In .scala.conf

scala.version = "2.13.6"

In .js.scala.conf :

include ".scala.conf"

scala.platform = js

Then all I need is to pass the configuration to the cli : scala-cli -C .js.scala.conf . and I have cross-compiled the project. It also sort of gives me the ability to easily switch from one target context to another in the editor, by structuring my config file accordingly and change a single "include" statement (which pretty much gets instantly taken into account by the editor after a ctrl-s in a scala file). That's a feature I'd really like to see in metals (https://gitter.im/scalameta/metals?at=611cfc533eec6d41d162026a), but scala-cli supports it "accidentally", and I think it's great that it does.

Dear Santa ...

With that in mind, and as a tooling-enthusiast, here's a list of things that I'd personally like seeing supported in scala-cli, as I think they'd open the door to really interesting integrations for library developers and medium/large application developers alike :

1. have a configuration setting for directories / files in the hocon format. The cli already supports passing directories, so I assume it wouldn't be hard to achieve, and it'd give users the ability to structure their directory a little whilst retaining awesome editor support, and have cross-compilation be further "accidentally" supported, by manually declaring version/platform specific folders in config files, and using include statements to combine them.
2. ability for the IDE integration to work with several directories that each have their own config file (ie as if it was a multi-module project). The cli already seems to use the configuration file in the folder it's being called, but the editor seems confused by it at the moment, and will only enable feature for one of the directories. I'm sure it's an oversight though, or a "todo" somewhere.
3. ability to declare "dependencies" over other directories (or config files), implying module to module dependencies. This is bound to be the most controversial point in this list, as it gets dangerously close to "build tool" territory, but considering how bloop allows for it and how scala-cli delegates to bloop, I wouldn't expect it to be drastically hard to support. It would bring interesting benefits, for instance allow for gradual separation of concern via strong semantic boundaries (by that I'm talking segregating dependencies in their own folders, coding via interfaces, etc) without having to rely on a fully-fledged build tool.

I think the integration of those three points would increase the potential of the tool without whilst keeping the scope focused, and would enable interesting integrations with mill / sbt or other tools, whilst offering the smoothest Scala editing UX that's ever been.

PS

I want to re-iterate how excited I am about this project. It scratches an itch that I've had for a long time and it already does it tremendously well despite how the project is in its relative infancy. Whether you see my feedback in a positive light, I'll 100% be using this tool (and already am, as a matter of fact).

[alexarchambault]

@Baccata Thanks for that lengthy feedback / proposal!

Some of the things you propose are planned, while our position right now is that others… are out of scope.

First, just so you know, we're not sure HOCON is here to stay. We have in the works a syntax such as this one:

using scala 3.0.2
using dep "org.typelevel :: cats-core : 2.3.0"

that can be embedded directly at the top of .scala files (it might still evolve a bit, and we may accept it as comments too, like // using … in a first time).

On top of that, we'd like to have something along those lines work:

require scala 2.12
require scala-native || scala-js

Just like using, these should be at the top of .scala files. (Also there's no consensus on the require keyword yet…)

using and require could be used in conjunction to enable your first use case, I believe. Users could have files such as

$ cat default.scala
using scala 3.0.2
using platforms scala-js, scala-native, jvm

$ cat js.scala
require scala-js
using dep "org.scala-js :: scalajs-dom :: 1.1.0"

In this example, the using directives in js.scala only applies to Scala.JS, because of the require in this file. (So overall, using are project-wide, while require are file-specific.)

We'd like to avoid directory-based filtering, that is reminiscent of Maven's src/main/scala or sbt's shared/src/main/scala-2.13. (It's something I'd like to clarify in https://github.com/VirtuslabRnD/scala-cli/issues/31, the gist of Scala CLI should be that it accepts a bunch of .scala files as input, and directory-based filtering goes beyond that IMO).

About 2., I guess you're using the setup currently described here? It might have issues when new files are created or not opened yet… You might have more luck with the setup-ide command, like

$ scala-cli setup-ide .

then running the "Connect to build server" command in VSCode.

About 3., it should indeed be the most controversial ^^ It's not in scope for now, and I wouldn't be in favor of it. As soon as you have several modules, the project is usually more complex overall, so it's nice to have an actual task system, for example to compile in parallel when possible, and be able to rely on its syntax to run specific tasks. And I'm not sure we should create or re-use one in Scala CLI itself. This also makes the inputs more complex than just a set of .scala files, so it would be out-of-scope for that reason too.

We don't have precise plans yet for how to go multi-module starting from a Scala CLI project, except a simple conversion of a Scala CLI project to sbt or Mill. https://github.com/VirtuslabRnD/scala-cli/issues/18 would be my preferred way of going multi-module, but it's not something we discussed much yet.

That said, we were also thinking of maybe allowing to integrate other builds in a Scala CLI invocation via BSP (or by extending it…) To allow for things like

$ scala-cli Foo.scala --add ../my-library

with ../my-library/.bsp describing how to start a BSP server to compile my-library, that could be used as a dependency in Foo.scala.

[tgodzik]

I don't think we want to work on that currently. I will close it, we can reopen if any new initiative is taken in this area.