Add language import for symbol literals

[dragos]

Let's add a language import that lets Scala 3 accept symbol literals. This would allow the large body of Spark notebooks to upgrade to Scala 3 with significantly less pain than otherwise, while staying out of the way for all other users. Spark already sets a number of predefined imports in REPL sessions (and implicitly, noteboooks), so this would be very easy to do and allow user programs to migrate painlessly.

While the Scala 2 compatibility mode allows this already, it's expected that most notebooks can trivially upgrade to Scala 3 for all other changes (as they are usually depending only on the Spark API).

Context:

• https://contributors.scala-lang.org/t/proposal-to-deprecate-and-remove-symbol-literals/2953/31
• https://github.com/apache/spark/pull/31569/files

[lihaoyi-databricks]

I think this would be a good thing to do, just to provide a more fine-grained control over deprecation of these old language features. I have no love for symbol literals, but these kinds of fine-grained compatibility flags are a lifesaver for migrating old code.

[smarter]

I'm not sure that's worth it, 2.13 already gives you deprecation warnings for using the symbol syntax and we should be able to remove something once it's been deprecated for years, especially when we have automated tooling to do these changes for you. Maybe the issue is that there's no scalafix for notebooks?

[dragos]

I had a quick discussion with @odersky before opening this ticket and he suggested we go this path. The motivation is three-fold:

• the contributors thread never concluded a viable path for the Spark body of Scala code, which is huge
• Scala 3 is a much nicer language for Spark than Scala 2, and most Spark user-programs could migrate with minimal changes. It would be boon to adoption if Spark would be an early adopter rather than at the tail end
• the glaring exception is the syntax column, which is used pervasively in books, examples and existing workloads. Notebook users are usually less sophisticated than regular programmers, notebooks often live in a "cloud", etc. It's a lot of friction.

The language import can sit quietly in the background and won't bother anyone. The quote is not any weirder than it is when used in meta-programming, and intuitively it can even be explained like foo is evaluated to its value, 'foo is evaluated to Symbol(foo).

From a purity perspective I agree with you. From a pragmatic point of view the cost of having this import is minimal, but the benefits are really big.

[odersky]

We have language.experimental.foo. Could there also be a language.legacy.bar?

[rxin]

To provide some context, 'foo is the DSL Spark chose to expression columns. That's used extensively within Spark's own code, in example code, and as a result by library/framework code and end user application code. The pull request @dragos linked shows that there are 3000 instances of use of the symbol in Spark's own code, and if you look up Stackoverflow, there are a large number of questions/answers with using' as well.

While it is a somewhat straightforward change, applications often depend on code that they don't control. Requiring applications and their dependencies to change the de facto preferred way of referring to columns (which is the most common expression) is not a great way to ask them to migrate to a new version of the language.

[mallman]

I'm not sure that's worth it, 2.13 already gives you deprecation warnings for using the symbol syntax and we should be able to remove something once it's been deprecated for years, especially when we have automated tooling to do these changes for you. Maybe the issue is that there's no scalafix for notebooks?

The most recent official distributions from spark.apache.org are built and run with Scala 2.12. Because of this, the deprecation and removal have caught some of us by surprise.

[smarter]

The most recent official distributions from spark.apache.org are built and run with Scala 2.12. Because of this, the deprecation and removal have caught some of us by surprise.

FWIW the spark situation was discussed back when this deprecation was proposed and one of the authors of Spark agreed that replacing the use of symbol literals by the existing string interpolator was a good solution: https://contributors.scala-lang.org/t/proposal-to-deprecate-and-remove-symbol-literals/2953/106, so maybe all that's missing is for Spark to deprecate all its methods that take symbols.

[rxin]

@smarter It was me who left that comment. The comment was about my personal opinion and preference. Regardless of what I believe, a large amount of code exists with 'foo and more will be written that way due to people copy pasting from past articles, stackoverflow answers ...

[SethTisue]

This seems like a good compromise to me — requiring the language import. It's similar to what we did with postfixOps. First we deprecated, then we required a language import too, then we removed it.

[SethTisue]

Note that https://scalacenter.github.io/scala-3-migration-guide/docs/incompatibilities/dropped-features.html will need updating.

[sarutak]

Hi, we are happy to have the language import feature for symbol literals. But symbol literals are still deprecated and removed in the future right? Also, is the language import feature supported at least within Scala 3.x?

[smarter]

But symbol literals are still deprecated and removed in the future right?

Yes, see https://github.com/lampepfl/dotty/pull/11588 for the details, in particular the documentation says "Features in this object are slated for removal. New code should not use them and old code should migrate away from them."

Also, is the language import feature supported at least within Scala 3.x?

No, this is an unsupported feature and we make no guarantee about how long it will stay in the compiler.

Closing since https://github.com/lampepfl/dotty/pull/11588 was merged.