scala / scala-dev

Scala 2 team issues. Not for user-facing bugs or directly actionable user-facing improvements. For build/test/infra and for longer-term planning and idea tracking. Our bug tracker is at https://github.com/scala/bug/issues
Apache License 2.0
130 stars 15 forks source link

Scala distribution replacement proposal #326

Closed adriaanm closed 3 years ago

adriaanm commented 7 years ago

By @dwijnand

This proposal seeks to address the problem of new users installing Scala, either with their package manager or with the distribution archives, and getting a setup that doesn't match how the majority of the Scala community consumes and develops in Scala.

The proposed way to address this is by:

  1. Removing the Scala distribution archives from the Scala website (and, therefore, from the Scala project's build and release setup and procedure docs.)
  2. Ensure sbt is appropriately highlighted and documented as the primary way the Scala ecosystem consumes Scala and builds their Scala libraries and apps, with sufficient detail on how to install and get started with sbt for it to replace the Scala distribution for new users learning and starting to develop in Scala.
  3. Make strides to ensure that there is a continuation path for the operating system-specific archives present in the Scala distribution, such as the deb, rpm and msi files generated.
  4. Provide a replacement solution for the legitimate users of the command-line runners in the Scala distribution (scala, scalac, scaladoc, and scalap).

Note that Scala will continue to be distributed as jars to Maven Central, which is how the majority of the ecosystem consumes Scala. A side-benefit of this change would, therefore, be that it would simplify, and thus reduce the maintainance cost of, the Scala project's build and release procedure.

A secondary action that is also proposed is to:

Background

Historically the Scala distribution has been available as: (in addition to api docs and sources archives)

Within the distribution are the Scala projects jars, and any dependencies:

as well as the following runner scripts, as both bash and BAT scripts, presented here with added description of what they allow users to do from the command-line):

with optionally the ability to define any:

Many years ago, when bootstrapping the Scala language and its subsequent ecosystem, that worked well. However, nowadays the majority of Scala users working on Scala projects will mostly never use the command-line runners or the bundled jars in the Scala distribution. A large number of Scala users will instead mostly interact with sbt, the most commonly used build tool in the Scala community, to:

A lot of tools in the Scala ecosystem provide an integration with sbt in the form of an sbt plugin which users consume by modifying either their build definition or their user-wide sbt settings. However, not every tool in a Scala user's toolkit is strictly part their project's build. So having every Scala tool be accessed through sbt is an avoidable imposition.

Unfortunately, within the larger JVM community there hasn't been any popular, general-purpose tool for downloading, launching, or bootstrapping tools, which has lead the ecosystem to repeatedly creating and maintaining a tangle of bash, BAT scripts and installation docs, with their related system properties, environment variables, and configuration files.

Within the Scala ecosystem, over time, there have been several, disconnected, projects that seeked to address how to launch, and possibly boostrap and/or install, command-line tools in the Scala ecosystems:

Some of the most popular tools built in Scala and used by the Scala ecosystem, and how they run, are:

Of additional importance sbt's launcher and sbt use a custom, non-standard classloader setup in order, for example, for the sbt launcher to be written in Scala 2.10 but run sbt 1 which is written in scala 2.12. See Jason Zaugg's notes for more details.

Notes and Design Decisions

Participants

Lead: Dale Wijnand (Scala contributor, sbt ex-maintainer)

I propose we invite the people we know in the community have exhibited an interest in this space, and might be interested in being involved:

Let me know if you have any other suggestions.

jvican commented 7 years ago

Interesting! /CC @heathermiller

antonkulaga commented 7 years ago

I personally even do not bother installing Scala, as sbt is enough to make Scala projects work and for REPL there is ammonite-shell

retronym commented 7 years ago

I've cataloged a few ideas for simplifying the SBT launcher, in particular around its classloader hierarchy. I'll work through these with @eed3si9n and @dwijnand, keeping the context of consolidation in mind.

jvican commented 7 years ago

@retronym I am actually working on a smarter mechanism to cache classloaders for sbt plugins, it builds upon the same concepts you describe in the catalogue. Will submit PR soon.

curoli commented 5 years ago

I routinely download the DEB or RPM files from the website to install Scala on all of my Linux machines. Is it really much work to provide those? I created DEB files of my own apps by SBT Native Packager and it has been quick and easy.

lrytz commented 5 years ago

What are you using the installed Scala distribution for? To do experiments in the REPL?

I definitely see the value of packing a Scala app as deb package so that it can easily be installed. But the Scala distribution is quite limited, as it ships only with the standard library.

curoli commented 5 years ago

Yes, experiments in the REPL.

dwijnand commented 5 years ago

Hey @curoli. It's not so much work, but it is work to maintain. What we're hoping to do is pull the deb/rpm/zip/tgz distribution building out of the build and replace them with (proposed) scala-runners project, which is based on coursier's app launching capabilities). Scala Runners would be directly consumable by users, but my thinking is that it could also be used by package maintainers if they want to re-build the deb/rpm packages (also the Homebrew formula for macOS and Chocolately/Scoop packages for Windows).

martijnhoekstra commented 5 years ago

@curoli have you considered https://github.com/dwijnand/scala-runners for your REPL needs (if you're not on Windows?)

martijnhoekstra commented 5 years ago

Jinx

dwijnand commented 5 years ago

(if you're not on Windows?)

Windows/BAT support is a requirement for scala-runners to be a replacement (opened https://github.com/dwijnand/scala-runners/issues/4)

ritschwumm commented 5 years ago

there's one thing the distribution gives you that none of the other ways of installation provide without glitches: you need connection to the internet exactly once to download it, and from then on it just works, regardless of whether you are connected or not.

with e.g. sbt it always feels like playing the lottery when youi try to develop on the train without connectivity: sometimes it works, sometimes it want to download new stuff and fails... those are the times where i pull our my trusty "normal" distribution and find something i can work on using that instead.

lihaoyi-databricks commented 5 years ago

I personally don't see the point in maintaining a "scala distribution" at all. Someone using Scala can get by just fine with SBT-only, or Mill-only, or Bazel-only, or even Intellij-only or Ammonite-only for small tasks. IMO just stopping the publishings of distribution and pointing people directly at the proper tools would be the greatest benefit for the least amount of work. Each tool already has good-enough integrations for scaladoc/scalac/console/etc.

I may be the exception in that I do not use SBT in any of my open source projects or work projects, but I do recognize it is widely used enough that for most people the SBT onboarding experience is as much a part of the Scala experience as the language itself. As long as it is the de-facto Scala experience, prominently advertising it and supporting it is the right thing to do

I like Coursier as an ivy replacement, but I don't think it's usage as an application-launcher and bootstrap system has been sufficiently proven out in the wild to label it as the "recommended" way of doing things. That opinion might change as coursier-as-a-bootstrap-launcher sees more adoption by people who aren't alex :P

dwijnand commented 5 years ago

there's one thing the distribution gives you that none of the other ways of installation provide without glitches: you need connection to the internet exactly once to download it, and from then on it just works, regardless of whether you are connected or not.

That use case is important, no doubt. My thoughts is scala-runners would support it and platform packages would use scala-runners to construct such "download one thing once" packages.

dwijnand commented 5 years ago

I like Coursier as an ivy replacement, but I don't think it's usage as an application-launcher and bootstrap system has been sufficiently proven out in the wild to label it as the "recommended" way of doing things. That opinion might change as coursier-as-a-bootstrap-launcher sees more adoption by people who aren't alex :P

Yeah, absolutely fair. I guess you can call me an early-adopter fanboy, but the viability of scala-runners depends on how well it works.

martijnhoekstra commented 5 years ago

Someone using Scala can get by just fine with SBT-only, or Mill-only, or Bazel-only, or even Intellij-only or Ammonite-only

I don't agree that this is a reasonable approach for quick REPL experiments.

ashawley commented 5 years ago

One solution isn't writing new Scala tools, but making Scala available on distribution channels that users are already using. The sbt-extras hack and the Apache Spark installation are embarrassing examples of how deficient things are. There's no reason sbt and Scala couldn't be first-class citizens on more distributions.

However, this will require the Scala community to follow the conventions and adopt the tooling of these other projects. The gap was evident recently when a Debian contributor was trying to work on getting Scala updated, but ran in to trouble.

https://contributors.scala-lang.org/t/question-about-building-scala-and-sbt-proper/2811

For example, if Scala and sbt were available on distributions like Debian or Fedora, it would likely trickle to Ubuntu and Red Hat and Amazon Linux, and therefore to the many Cloud, serverless, container orchestrated, or Continuous Integration services. The work with Scala could also pave the way for packaging other Scala tools like Coursier, mill or Spark.

Scala and sbt have long development cycles, so once the pipe is primed, things could work pretty well. As evidence, consider the Macports and Homebrew packages for sbt. They stay updated, and are very reliable in my experience.

Most free and open source projects know this, and bake in these guidelines from the beginning. Scala should, too. It doesn't help that the underlying Java ecosystem sometimes doesn't have the best story here, either. It's getting better, though. Scala should catch up, as well.

dwijnand commented 5 years ago

I don't agree that this is a reasonable approach for quick REPL experiments.

@martijnhoekstra I agree and it's why I built Scala Runners. Have you given it a go? It needs improvements, so I'm looking for feedback. 😄

martijnhoekstra commented 5 years ago

@dwijnand due to dayjob reasons, I run Windows, and after not getting it to work for, like, 5 minutes, I gave up. In theory, it's just what I want though.

@ashawley IMO a standard scala distribution for more distributions is not as useful as it sounds. You are going to want to use a build tool for anything but repl'ing anyway (and repl experiments, at least for me, are often version dependent) and the build tool will download it's own version. I'm struggling to find a use case for it.

A scala library package is even less useful IMO -- linking against a static scala library jar on a fixed classpath is just not how applications tend to run.

a scala build tool like sbt, cbt or mill is much more reasonable to want as package IMO. The bootstrap fo such packages is an issue though, at least for Debian.

ashawley commented 5 years ago

I don't disagree. I even mentioned sbt, mill and Coursier in my comment. Even Adriaan's original proposal mentions supporting "legitimate users of the command-line runners in the Scala". We need to provide a lot of different avenues for people to use Scala, though. We need to argue less about picking a single solution.

curoli commented 5 years ago

It's really very simple. If I want to install Scala, I want a package called scala that I can install with apt or dnf, and if I want to run Scala, I want a binary called scala that I can run in my terminal or find in the app finder. That's just The Normal Way. I'm fine if package and binary are called scala, but it is actually Ammonite.

And if I hear that providing a distribution The Normal Way is considered to much maintenance work, then I am very worried, because I don't see why anything else will be less maintenance work, and I don't want to trade The Normal Way for something I need to unzip or clone from GitHub, or something that first needs to do some downloading or setting up before it can run.

Yes, I'm using SBT and IntelliJ for development, and that doesn't need the scala package, but I still want my scala to play with and I want to have it The Normal Way.

(And for the record, yes, I totally hate it that JetBrains doesn't provide proper packages for IntelliJ and instead expects people to download and unzip for every major version.)

dwijnand commented 5 years ago

I want to run Scala

Personally, I think that sentence doesn't make sense. Scala is a project and not an app or a specific command-line tool, so but I don't think "run Scala" makes sense.

There's also no "javascript" package and binary, nor a "haskell" package and binary (I'm using Homebrew as my basis for these claims, sorry for the bias). There is a "rust" package but it doesn't give you a rust binary. It gives you a rustc and a cargo binary, and they work in concert on the PATH (much unlike sbt and scalac).

There was a version of this proposal that intended to replace the contents of the Scala distribution with sbt (as the 80% case) and coursier (as the tool to get the other 20%, or at least one way to, that you get "out the box"). But when I thought of the counter-examples I mentioned above I realised it's better to just let sbt and coursier do its own distributions and advertise them more/better in scala-lang.org, and spin out a "scala-runners" project for the command-line tools (with a modern, coursier, take on it).

curoli commented 5 years ago

Of course, scala is an app. Let me show you:

oliverr@GP7E3-45D:~$ scala Welcome to Scala 2.13.0 (OpenJDK 64-Bit Server VM, Java 1.8.0_212). Type in expressions for evaluation. Or try :help.

scala>

See?

ashawley commented 5 years ago

It's really very simple. If I want to install Scala, I want a package called scala that I can install with apt or dnf, and if I want to run Scala, I want a binary called scala that I can run in my terminal or find in the app finder.

Yeah, I also wish that Scala tools would be easier to install with existing package tools. I think if Scala and sbt get the ball rolling, then it wouldn't be hard to make way for Ammonite or any other host of tools. The Node, Ruby and Rust communities know this. Their language-specific build or package tools are also there. And even though it's not always likely that you need to run their interpreters or compilers directly, you still can.

curoli commented 5 years ago

There is no "javascript" package, because JavaScript runs in your browser. I don't know what's up with Rust or Haskell, but I don't personally know anyone using those. Popular languages used for scripting like Python and Perl follow, AKAICT, The Normal Way. So does Ruby.

dwijnand commented 5 years ago

Let me show you

When a seasoned Scala developer says "I need to set up my machine for Scala development" or a Scala beginner says "OK, I want to learn Scala" do you think that's what they're after?

To me for the former it's obviously no, and I'm not sure that it's yes for the latter either (I'd love to hear more feedback from people teaching Scala to beginners on this).

My thought process is that if there isn't a Scala package then, for beginners, after the initial failed apt-get/yum/brew/chocolatey/scoop install scala they'll go to https://www.scala-lang.org/ and find https://docs.scala-lang.org/getting-started.html and discover the various options on how to get set up and get started learning Scala.

WDYT?

ashawley commented 5 years ago

Yes, that's true. People who want to learn Scala need to be course corrected to use Intellij and sbt. That is how I would summarize the web site. In addition to informing beginners they need sbt, not scala, it would help if they could use apt-get/yum/brew/chocolatey/scoop install sbt.

som-snytt commented 5 years ago

Now I use sdkman to select my java and my scala (release versions). I have crummy old tools at the office where I have to use old java (meaning 8) before launching some tool. I am normally on Ubuntu and avoid actual "package managers" because of the dance around removing packages before adding them, etc.

As a footnote, I had to buy a windows box for work, then was excited about the linux subsystem, but that has proven to have drawbacks. For instance, I often launch an Ubuntu shell but then it has trouble seeing keyboard input.

In a word, alongside actual solutions, there are many non-solutions.

ritschwumm commented 5 years ago

seconding som-snytt, for java and scala i usually just download the zip/tgz, unpack it somewhere in my home and set the PATH. "proper" OS packages tend to be outdated to a varying degree. no need for debian packages or similar.

curoli commented 5 years ago

som-snytt, ritschwumm, that typically requires a lot of work to figure out, which dependencies you need to install, or upgrade, and even more work later, to figure out which ones can be removed again.

True, the packages made available by popular Linux editions are outdated. E.g. Ubuntu 18, currently offers Scala 2.11.12. But, if you have a DEB file for a newer Scala version, your native package manager will gladly manage dependencies, install, upgrade or remove.

neontorrent commented 5 years ago

What are you using the installed Scala distribution for? To do experiments in the REPL?

I definitely see the value of packing a Scala app as deb package so that it can easily be installed. But the Scala distribution is quite limited, as it ships only with the standard library.

I personally use it as an advanced calculator very often. I work in a financial field and often loads up a library and do calculation without creating a new project every time.

Yes, that's true. People who want to learn Scala need to be course corrected to use Intellij and sbt. That is how I would summarize the web site. In addition to informing beginners they need sbt, not scala, it would help if they could use apt-get/yum/brew/chocolatey/scoop install sbt.

I wouldn't think forcing users to use a "corrected" way will promote Scala nor it is friendly to new users.

REPL is quite important and useful when someone who just came to the language and freely doing experiments without getting into the big giant ecosystem of sbt or whatevs.

dwijnand commented 5 years ago

REPL is quite important and useful when someone who just came to the language and freely doing experiments without getting into the big giant ecosystem of sbt or whatevs.

I agree the REPL is important and it has users, and so there will absolutely need to be a solution to continue to support such users and use cases.

But, sbt is also important, and the current distribution biases to providing the REPL and a command-line interface to scalac and scaladoc (and a runner for launching classfiles/jars/scala code, and niche scalap, and deprecated fsc).

So how do we address that? What do you suggest?

I wouldn't think forcing users to use a "corrected" way will promote Scala nor it is friendly to new users.

Could you expand on why you think so? (Both the promotion and friendliness parts.) Thanks!

curoli commented 5 years ago

Let's say you want to try out a new language you don't know yet. How would you prefer to do that? Would you rather start a shell where you can try out a few expressions? Or would you rather set up a build tool that you don't know yet, either?

dwijnand commented 5 years ago

I have my personal preference, but that shouldn't be the decider. I'd love to hear from people teaching Scala (Martin, other professors, teachers that use Creative Scala, MOOC teachers, etc) on what they've found works best.

To add to the REPL vs setup build tool options, there's also running Scala in the web and setting up Scala in an IDE. These are all detailed in https://docs.scala-lang.org/getting-started.html.

The fact that they are all described in https://docs.scala-lang.org/getting-started.html to me means that they're all equally likely and liked. Which is why I think we should de-emphasise the REPL (but certainly not make it hard to get, if desired!)

julienrf commented 5 years ago

I think the simplest getting started experience is to use an online tool such as scastie. But that’s probably out of the scope of this discussion, which focuses on the next step: how to simplify the installation procedure to manage Scala projects locally? I guess we should have a tool that could be installed in one step and which would then manage the installation of the other tools mentioned in the issue introductory post (scalac, scala, sbt, ammonite, etc.). Similarly to what pip does in the Python world, or npm in the JavaScript world.

dragos commented 5 years ago

The fact that they are all described in https://docs.scala-lang.org/getting-started.html to me means that they're all equally likely and liked.

I wouldn't take that page on the website as proof to what people like.

Having taught a bit of Scala to both students and developers, there is no doubt as to what is easier: the REPL wins hands down as first-contact with the language. Sbt is extremely tough on newcomers, especially when things go wrong and pushing it more in front of beginners is something I would avoid at all costs. Of course, once someone decided to start a project in Scala, Sbt is probably the right choice, but that's not how we want the first contact with the language to look like.

Java finally added a shell (instead of directing people to Maven), so I would take that as a data point that a REPL is a good way for people to interact with the language, and in my experience it is still the best for beginners. I use the REPL all the time, and so do most people I work with. I almost never run sbt console, simply because I may need to start Sbt (~15 seconds), or it wants to build the current project (if I need the REPL it probably does not compile anyway).

I was under the impression that the distribution was largely automated, so the cost of providing at least a tarball is rather small.

dwijnand commented 5 years ago

You want sbt consoleQuick if you don't want the project's classes.

neontorrent commented 5 years ago

I think the sentiment is people want more options instead of less, and a few people here including myself have expressed our use cases.

I would suggest that we provide distributions with

and let the users decide what they want?

som-snytt commented 5 years ago

Just to note that users don't know what they want, they just want it to work in whatever context they happen to find themselves. I am one of those users.

Edit: to clarify, in the Steve Jobs sense that the visionary provider has to show the way to an outcome that the user could not foresee. I know what I want only in the sense that I know what I do: I download the tarballs for REPL and one-off tests and scripting, and I use sbt, mostly. But, for example, sdkman made a big difference in my workflow for jdk switching. Once upon a time, jdk switching was rare for me. sdkman also mostly works for scala. But there is surely a better way, if only I knew what it was. So I kind of know what my use cases are (but even they are ripe for disruption), but I don't have a visionary sense of the way we're all doing it wrong. Unless we go back to sbaz, which I never experienced first-hand, but I think we should migrate sbt to sbz.

Jasper-M commented 5 years ago

IMHO scala-runners—if it works well—seems to strike the perfect balance between having a native binary and going through sbt to launch a REPL.

neontorrent commented 5 years ago

Just to note that users don't know what they want, they just want it to work in whatever context they happen to find themselves. I am one of those users.

I had a different experience. I know which distribution to get and what to do with it. Other devs in my team have the same experience so I wouldn't say users don't know what they want.

IMHO scala-runners—if it works well—seems to strike the perfect balance between having a native binary and going through sbt to launch a REPL.

runner

That tool looks very nice. I am okay with it if it is the official solution. Does it have dependency management? Sorry I might have missed it because I don't see in the README

dwijnand commented 5 years ago

No, it doesn't. But we've started a conversation about that in https://github.com/dwijnand/scala-runners/issues/3.

eyalroth commented 4 years ago

The proposed way to address this is by:

  1. Ensure sbt is appropriately highlighted and documented as the primary way the Scala ecosystem consumes Scala and builds their Scala libraries and apps, with sufficient detail on how to install and get started with sbt for it to replace the Scala distribution for new users learning and starting to develop in Scala.

Please don't; or rather, please advocate the other tools as well.

It's been a while since this ticket was opened, and in that time there have been multiple ocassions where the question of what is Scala's primary build tool is -- or should be -- has been brought up:

  1. Scala Users: Who is winning the build war with Scala So Far? (2019)
  2. Scala Contributors: Why does Scala need its own build tool (SBT)? (2019)
  3. @lihaoyi blog: So, what's wrong with SBT? (2017) (see comments and reddit discussion as well)
  4. YCombinator discussion following "Towards Scala 3" blog post (2018)
  5. Linkedin blog: Productivity at Scale: How We Improved Build Time by 400% at LinkedIn (2018)

I personally am greatly in favor of Gradle. It's more mature, more popular, better maintained, and most importantly - more familiar to Java developers, which are the potential adopters of Scala. Only downside is that it's currently using "the old Zinc", but integration with "the new Zinc" or Bloop is one of the most voted tickets in their repository and on the milestone of their next release (6.0).

bjornregnell commented 4 years ago

I teach Scala first at Lund University. Compiling in terminal with scalac and see that class-files are created is a second step after gentle scala REPL experiments and before using a build tool. It's good for your conceptual understanding to see what's going on with a one-file object Main and input-output of the compiler.

My students want sudo apt install scala on ubuntu (dual boot or vm with virtualbox is common) to get the REPL and scalac. And many look for scala in app stores on windows and macos and ask me why its not there...

lrytz commented 4 years ago

(I think) we are not suggesting to kill the scala and scalac command line tools, just change the way they work. Today you have to download the distribution (jars + scripts). In the new model, the commands would be scripts / aliases for coursier commands / coursier apps. How exactly you'd install them is to be discussed, but I think it would be easier for us / for contributors to provide a good experience across platforms than today.

bjornregnell commented 4 years ago

to provide a good experience across platforms than today

Yes, that would be really nice! Something to write in terminal and then scala and scalac just works, and I would like to have sbt as well in the same "package" deal. (cf the magic brew paste which is a bit cryptic but works). But a person using a platform expect something typical for that platform... That's why I also suggest a "scala app" in the respective main stream app store (apt+snap, ms store, apple app-store).

Jasper-M commented 4 years ago

I don't think you'll find many compilers or related command line tools which are available via the actual app stores.

bjornregnell commented 4 years ago

No, but if so it would make it easy to install'em. Programming für alle :)

neontorrent commented 4 years ago

No, but if so it would make it easy to install'em. Programming für alle :)

I agree. Improving distribution can help newcomers to quickly get hands on. This is definitely something we can do.

Pretty much the only dependency for Scala is JRE. Many linux distributions now have repo with OpenJDK. We can add Scala distribution to the repo and add OpenJDK as dependency.

eyalroth commented 4 years ago

I don't think you'll find many compilers or related command line tools which are available via the actual app stores.

I'd say that the vast majority of software is not available via the app store in the desktop world. Well, at least not on Windows.

The biggest exception that I can think of is games, which are usually available via a game store launcher (Steam, Uplay, Origin, Epic, GoG, etc), but they are an entirely different beast (paid, online multiplayer, extremely large download size, etc).