Support Scala 3/Dotty

[Kazark]

Dotty, the Scala 3.x compiler, is beginning to mature. It is supposed to drop around the end of 2020. The community is poised to hop onto it pretty fast, and some libraries are already releasing Scala 3 versions. We'll need support for it in Emacs at some point.

One of the questions is, do we pretend that Scala 2 & 3 are really the same thing? I.e. do we just mash all the features together? Or, do we have some kind of switch by which you can set your Scala version/we detect it from SBT/other? There are syntax changes, but also semantic changes. The website linked above has a good amount of documentation.

I'm happy to help with this effort if needed. I am still relatively new to Emacs but learning all the time.

[hvesalai]

Personally I haven't yet looked much into Dotty.

Is there any abstract or concise presentation of the syntax changes? I think whether we pretend that Scala 2 and 3 are the same thing or not depends on how many and how large the changes are.

[hvesalai]

The current version of scala-mode was implemented against the Scala Language Specification 2.11, which had a nice BNF. Is there a BNF for Dotty?

[hvesalai]

And to answer my own question, yes there is https://dotty.epfl.ch/docs/internals/syntax.html

[Kazark]

Opened https://github.com/hvesalai/emacs-scala-mode/pull/161 as a simple starting point.

[Kazark]

Here's the next thing I'm encountering, after the PR above. Because the enum { ... } syntax for declaring ADTs is new, but the case keyword is not, Emacs is getting confused about where to put the cursor when I start a new line. I'm getting tabbed in too far:

I'm investigating this but it's going to require me to understand the code more deeply than the first PR. Pointers would be appreciated.

[Kazark]

I'm thinking scala-syntax:newlines-disabled-p is the right place to start looking at that problem.

[Kazark]

Wow, I am going to have to understand a lot more before I'm going to know how to change this function. I've got the general idea of what's going there but I'm not close yet on grokking it enough to feel safe changing it.

[hvesalai]

Yeah, you need to understand the logic of the indentation engine. But you are correct, that's the place to check if the case is inside enum block.

[hvesalai]

Here's the things you should understand:

1. syntax-ppss is a function that gives you the parser state at a given point point, which is the current cursor point, unless otherwise specified. The result is stored into a variable called state
2. the (nth 1 state) returns the first element of the parser state, which happens to be the position of the block opening curly brace. This is stored into a variable called parenthesisPos
3. the line (goto-char parenthesisPos) jumps to that point

So after that statement you are in a position where you would have { if you were in an enum block. Also the first non-comment or whitespace token left of that position would be enum. So what you have to do here is to check that it is so. You can do that by first calling scala-syntax:backward-sexp and then checking if the thing at point is enum by calling (looking-at scala-syntax:enum-re), where scala-syntax:enum-re is a new variable you have defined (see scala-syntax:case-re for an example on how to do that)

[Kazark]

@hvesalai Thanks so much for the explanation! That jives with what I was beginning to understand, but fills in some really critical details. In fact, that's probably all I need to get this working. If I get it working, I'll open a separate PR (unless you want me to bundle it in with my other one and just start to make one large Scala 3 PR).

[hvesalai]

Make separate small PRs where possible.

[datalek]

Hi! I'm also interested to have the support to dotty. Probably I'm missing something (sorry, I'm pretty new to emacs) but should we get the support to it pretty easily thanks to emacs-metals and lsp-mode module?

Can I help somehow to get it available?

[hvesalai]

@fommil can you give your educated guess about this topic? I know you know a thing or two about it. :-)

[fommil]

me, when people ask how easy it would be to retrospectively add significant whitespace to an Emacs major mode:

I've written a major mode from scratch with a significant whitespace indenter, and it was a lot of fun to learn all the Emacs APIs and I learnt some lessons too about how I'd do it the next next; but I think I'd need a really strong reason to go down that route, e.g. Scala 3 completely dominating over Scala 2 and me needing to use it for my job. I'm still unsure if Scala 3 will see any serious industry adoption beyond open source hobbyists and EPFL academics.

[hvesalai]

Uh-huh. I didn't know that Dotty has significant whitespace and I'm shocked to hear it has. What the ... were they thinking? I'll just bury my head in the sand and hope it goes away.

[Kazark]

@hvesalai not quite sure how to take this. Does this mean that I should fork this to develop a separate Scala 3 package? I work for a major enterprise that uses Scala heavily and is poised to move rapidly to Scala 3 once it becomes fully available.

[hvesalai]

@Kazark Not all. My comment was more meant against the Scala stewardship in this matter.

The question I have is, can scala 3 support be added to this project without breaking it for scala 2. If it can, then let's do it in this project. If not, then we need to make a decision: do we start maintaining two branches in the same project (with releases being made from each), or do we create a completely new emacs-scala3-mode.

[Kazark]

Ah, I see. I think Scala 3 is backward compatible enough that if we targeted it, it should basically cover Scala 2 as well (though perhaps breaking it in the sense of making things that are illegal in Scala 2 appear legal).

Let me know how I can help. I don't know much about major mode development but I'm willing to learn. This is important to me for my job, so I am happy to help how I can.

[hvesalai]

Major-mode development is not that special and the mode I've written already has all the interaction you need in it, so you don't have to learn that side. What you will want to have is the basics of elisp.

Then it's just a matter of getting acquainted with the code base. If you can take a good look at the files first your self, we could then have a online (video) meeting where I can introduce you to the code base by walking through it and maybe simulating with you some functions (i.e. how the program flows through the different functions).

[Kazark]

@hvesalai I do have a workable knowledge of ELisp. I'll work on familiarizing myself with the codebase.

[fommil]

I think Scala 3 is backward compatible enough that if we targeted it, it should basically cover Scala 2 as well

I don't want you to be under any illusions; what you're planning to do here is a major undertaking unless your goal is modest limited to just a couple of new syntactic structures. You'll find that the new parts of Scala 3 are not backwards compatible and you'll need to make a decision about how much can be supported (e.g. we wouldn't want new Scala 3 keywords to become treated as such in Scala 2 when they are valid identifiers, and vica versa like do / while which was removed).

The Elisp manual has great documentation on syntax tables, fontification, tokenisation and indentation, so I recommend starting there. Although I think the indentation strategy used by scala-mode is "classic" compared to the more modern recommended approaches (e.g. SMIE). The impact of significant whitespace on indentation cannot be underplayed.

[Kazark]

we wouldn't want new Scala 3 keywords to become treated as such in Scala 2

Unless I misunderstand you, it's too late on that already, after #161 .

[Kazark]

Whew, I begin to see why this could be a big job.

[hvesalai]

Is there a specification of the new syntax in some common format (BNF, etc)?

[SethTisue]

Is there a specification of the new syntax in some common format (BNF, etc)?

https://github.com/scala/vscode-scala-syntax might be helpful here

@nicolasstucki is there anything else that contributors here ought to consult?

[smarter]

The formal grammar is at http://dotty.epfl.ch/docs/reference/syntax.html, and http://dotty.epfl.ch/docs/reference/syntax.html describes how the indentation-based syntax works.

[smarter]

And more prosaically, here's the regexp I wrote to determine whether indentation should be increase when pressing Enter at the end of a line: https://github.com/smarter/vscode-dotty-syntax/blob/153103a27f3fcb123f320e9f7b53548e8d07881e/src/extension.ts#L31-L36 (currently in a separate extension until https://github.com/scala/vscode-scala-syntax/issues/179 is fixed).

[Kazark]

It's good I didn't jump into this when I first submitted the issue. I see the syntax has already been changed. Presumably it is now stable since the release party is scheduled for Apr 23.

[Kazark]

Spent some time reading the code last night. Made some progress in understanding it.

[hvesalai]

Ok. Do you want to have a code walk through session at some point? We could set up Google Meet or anything that works for you.

[Kazark]

Sure, that sounds good. Maybe send me an email at kazark@zoho.com and we can set something up privately that way.

[Kazark]

Unrelated to Scala 3, but maybe I can fix it while I'm in there. I noticed we don't always handle trailing commas well, e.g.

def foo(
   param1: Type1,
   param2: Type2,
   param3: Type3,
): Foo = ???

The ) gets indented inappropriately:

def foo(
   param1: Type1,
   param2: Type2,
   param3: Type3,
   ): Foo = ???

[hvesalai]

The trailing comma handling is based on what scala 2 has originally accepted (I don't know if it does not, but it didn't use to accept trailing commas).

[Kazark]

Yeah, I don't remember what version that was that changed it, but maybe 2.12 or 2.13.

[fommil]

I think you might have found the only tool in the entire ecosystem that @dwijnand didn't update to support trailing commas.

[dwijnand]

It changed in 2.12.2, btw 😄

[Kazark]

In case anyone wants to track my progress or comment as I go along (or wonders whether I am actually working on this), I've opened a draft PR: https://github.com/hvesalai/emacs-scala-mode/pull/170

[sideshowcoder]

@Kazark I see you are regularly updating the branch, does it make sense to start testing this? I ran into Scala 3 syntax recently and the best I could come up with is to turn off the indention for now but obviously this isn't a great place to be.

[Kazark]

@sideshowcoder great question. I keep wanting to call for testing, and then I hit a different my scenario myself that behaves really, really badly. After my last set of changes, a lot of the indentation was working way better... but now it seems I can get into an infinite loop in some cases 😬 I need to fix this soon as it is definitely tripping me up in my own development. If I can get it to where I can use it for a week with nothing egregious, I'll ping here and ask for testing. Thanks for the question.

[smarter]

but now it seems I can get into an infinite loop in some cases

Might be unrelated but there's one open PR which fixes a not-quite-but-seemingly infinite loop on master: https://github.com/hvesalai/emacs-scala-mode/pull/157

[Kazark]

Okay, thanks. I tracked it down and in this case it was entirely self-induced.

[Kazark]

@sideshowcoder you could give it a try now and see if it puts you in a better place at all. Let me know any problems you hit.

[prassee]

any update on this ?

[Kazark]

@prassee I've attempted to hand this work off to someone else, but don't know whether he's made any more progress on it. I certainly haven't.

[maemre]

@prassee I am that person @Kazark handed it off to. I made some progress (my progress so far on top of @Kazark's work is here). Most of it was disabling and adjusting some heuristics, and fixing some small bugs. I haven't made much progress since April (I was dogfooding and documenting the bugs I encountered, but I didn't find the time to fix the bugs). It is mostly usable but there are still some noticeable issues around extension, case, do, end and a few other places.

I'll submit a PR to @Kazark's scala3 branch once I clean up my additions, remove debug messages, and fix at least easy bugs (which should be done by ~this weekend).

[KaranAhlawat]

Since this issue is still active in some capacity (as in people come and look at it), I'd like to add that I'm working on a scala-ts-mode, based on the treesit support in Emacs 29+, and of course, any help is appreciated.

Maybe it'd even be better to focus efforts there? @maemre @Kazark

[udalrich]

Is this still active? I'm trying to use the new syntax at work, and not having auto-indentation work is annoying.

I'm running version 20221025.1502 from (I think) elpa, so if the fixes were merged, they don't seem to have been released.

If it needs more work, I'm willing to contribute

[KaranAhlawat]

@udalrich Well, there is scala-ts-mode depending on what version of Emacs you're running.