search

twitter / util

Wonderful reusable code from Twitter
https://twitter.github.io/util
Apache License 2.0
2.68k stars 580 forks source link

Path to Scala 3 #274

Open felixbr opened 4 years ago

felixbr commented 4 years ago

This is an issue for discussing and tracking necessary step to prepare for a Dotty/Scala3 cross-build.

Inspired by the awesome "Scala Love" conference, I attempted to cross-build util-core with Dotty 0.23.0-RC1. So far it went smoother than expected. In the end there were about 4 compile-errors that I couldn't directly address.

In any case there's some work that has to be done before a real cross-build can be achieved. So far I've identified the following but I will try and update this list as we learn more.

Preparations:

Resources for Dotty migration/cross-building:

I hope you don't mind this type of issue. This is not meant to push anyone, just as a place to discuss and organize things.

Cheers ~ Felix

mosesn commented 4 years ago

@felixbr yep, we're perfectly happy to have those kinds of conversations here. If it turns into a big effort, we'll probably set up a ticket like https://github.com/twitter/finagle/issues/771 for doing it. We're going to upgrade scalatest soon to support https://github.com/twitter/finatra/pull/527.

I can file an internal ticket around upgrading our sbt version, but we're going through some turbulence with our builds, so there's a good chance that we'll have to hold off on that until we've worked out some of the details.

mosesn commented 4 years ago

Oh, also, as we find things that are challenging to migrate, that's probably good feedback for the scala compiler team. Especially if there are things that scalafix handles poorly. Please let us know what your experience in trying it out is like! Thanks for taking this on, I'm excited to see what you find =)

cacoco commented 4 years ago

@felixbr FYI, we've updated to sbt 1.3.10 in 6ee7b49a7aa22462648dd9be27e2c577b3a9f003. Scalatest upgrade may take a bit longer. Thanks.

mosesn commented 3 years ago

@felixbr sorry for the delay, scalatest was a beast. It's finally updated https://github.com/twitter/util/commit/db63a6dcf2113606f27e04d6860ad2d0ca72ab5e

felixbr commented 3 years ago

Oh, that's great news! :)

mosesn commented 3 years ago

@felixbr what do you think are the next steps to supporting scala 3? We've tackled the two you suggested before.

felixbr commented 3 years ago

The biggest problem is that Dotty/Scala 3 is not stable yet, so (if I understand correctly) for any given Dotty version we want to use for a cross-build, the dependencies have to be available. It's a moving target and that makes it difficult at the moment.

I just looked at maven-central and it seems that for Dotty 0.23 a Scalatest 3.1.2 version and for Dotty 0.24 a Scalatest 3.2.0 version exists. The latest Dotty version is 0.25 (released 5 days ago).

My guess is that moving to Scalatest 3.2.0 could be a lot of work again, so for the moment we could attempt the cross-build with Dotty 0.23 but I have no idea how outdated this is compared to 0.25.

Maybe that's something we should clarify with the Dotty-Team before we attempt cross-building with an older version.

mosesn commented 3 years ago

@felixbr I'm inclined to just start with 0.23, and we can move to newer versions after we upgrade our scalatest. I think scalatest 3.1.x's scalafix rewrite will be easier to use than the 3.0.8 one, so I'm cautiously optimistic that the 3.2.0 upgrade won't take that long.

@SethTisue what do you think is the right way forward here?

mosesn commented 3 years ago

Poking around a bit, it looks like @martijnhoekstra also tried porting util to dotty, and ran into some issues: https://contributors.scala-lang.org/t/scala-3-migration-guide/4237/6

What is the usual strategy that people take? Do they keep two different sets of code, or do they use the subset of scala that is supported by both scala 2 and scala 3? I think one approach I would be happy with would be to keep developing in scala 2, and then have a tag that continuously tracks "develop but scala-3-ified"

martijnhoekstra commented 3 years ago

@mosesn to me it made sense at the time to report the encountered issues to dotty and not bother with workarounds yet, since introducing (sometimes ugly) workarounds for bugs that probably won't be in scala 3.0 release didn't make sense to me.

IIRC the biggest issue was the dropped support for macros which is used in IIRC the deprecated Configuration stuff (with existing notes about breakages). Apart from that, cross-building with dotty seems plausible on a unified codebase without any scala-2/scala-3 specific stuff.

Until scala 3.1 rolls along that is, but it seems too early to worry excessively about that.

mosesn commented 3 years ago

Cool! I'm glad to hear it. Which Configuration stuff? Util doesn't have any macros afaik

martijnhoekstra commented 3 years ago

I took another look @mosesn

It wasn't macros, but it was a dependency on scala-reflect in Config (filtering out synthetics).

Apart from that, for now the dealbreakers look like the same as mentioned on the contributors thread: no @varargs support just yet (so no java-compatible varargs forwarders and no overriding of java-compatible varargs, needed in InMemoryStatCounter) and no good story for cross-compiling dynamic member selection (on unsafe in jvm/HotSpot)

martijnhoekstra commented 3 years ago

https://github.com/twitter/util/compare/develop...martijnhoekstra:dotty-compat is a quick port -- in this state all main projects compile (but test crash-and-burns)

mosesn commented 3 years ago

Got it. What do you think would be the right way to interact with the dotty team to get them to prioritize this stuff? It would be really neat for us to be able to add util to the dotty community build.

felixbr commented 3 years ago

@martijnhoekstra I found pretty much the same blockers when I attempted the cross-build. I didn't really understand what exactly scala-reflect was used for. Do you think we can just remove it?

Maybe we should gather the missing features and then talk to the Dotty team, which of them can be expected to be solved by them and which need a separate workaround anyway.

Are these all blockers right now (apart from tests, of course)?

In general I'm in favor of trying a unified codebase first even though writing code that works on 2.11 through 3.0 will be interesting :smile:

martijnhoekstra commented 3 years ago

I bet it would be neat for the dotty team too, I think 2.13 would have been smoother if we ported earlier in the RC cycle too and it would be a shame if that happens here. Now is the time to do that.

The dotty team is really responsive to issues. They merged fixes for issues I opened earlier in a matter of hours after reporting. I can check tomorrow what issues are still open and give them a ping and then we can open new tickets for what's still pending.

Then maybe link them here, so it's easy to keep track what workarounds can be undone/what blockers there still are, so you can give that a nudge every now and then if it doesn't move.

Getting scalatest working and getting CI going for this repo will also help. I'm not sure what the plan is for the scalatest people, whether they'll release everything to 0.26 or just the latest version. I see they have a PR open for 0.26 but I don't know how soon they plan on releasing that.

Moving this repo to scalatest 3.2 was kind of annoying, but not too bad, even without the scalafix they have. If you still need to upgrade dependencies company-wide, then there isn't too much use in a PR for that I would think?

larsrh commented 3 years ago

I have a bit time at my hands. Would you be interested in me sending a PR porting to ScalaTest 3.2? (I think this is a good idea regardless of Dotty.)

martijnhoekstra commented 3 years ago

Reflective access is fixed in https://github.com/lampepfl/dotty/pull/9420 -- it's not released yet. To workaround, we could maybe use a scala-2 specific shim. That probably needs some sbt-foo to get a scala-2 only directory analogous to the 2.13+/2.12- directories. Or is that built-in in the dotty plugin?

The thing with scalatest is that (at least it used to be the case that) dependencies have to be updated twitter-wide, so it's broader than just this repo.

larsrh commented 3 years ago

The thing with scalatest is that (at least it used to be the case that) dependencies have to be updated twitter-wide, so it's broader than just this repo.

Yeah, @felixbr told me as much. Based on a quick investigation it also requires

martijnhoekstra commented 3 years ago

What I thought was a varargs issue now looks like an overriding bug in dotty: https://github.com/lampepfl/dotty/issues/9463

mosesn commented 3 years ago

I'm talking to a team in a couple hours that might be willing to move Twitter to scalatest 3.2, let me get back to you. If they don't have bandwidth, I can (probably) do it myself.

Re: Config, I think we can delete it, it has been deprecated for 7 years. That'll be an action item for me, we have to make a few changes in our codebase.

martijnhoekstra commented 3 years ago

That leaves the varargs overriding above that I haven't yet found a workaround for, and scalatest 3.2 actually getting released for dotty. The scala2 shim for reflectiveSelectable works fine, at the cost of some sbt config and a scala2-specific source directory.

felixbr commented 3 years ago

I've updated the issue with the action items you talked about and the two external dependencies we're waiting for.

edit: This time for real, the first edit got lost somehow...

mosesn commented 3 years ago

The meeting we had yesterday wasn't fully resolved, I'll take a stab at applying the scalafix patch (it's complicated because of the scale of our codebase) and see how it goes. Unfortunately our scalafix experts are on vacation, but hopefully I don't run into anything too bad 🤞

martijnhoekstra commented 3 years ago

I asked about the dotty 0.29 plans of scalatest on their mailing list, but my message is still in moderation.

martijnhoekstra commented 3 years ago

I see there is a PR open now to bring 0.29 support to scalatest 3.1.x which I seemingly prematurely assumed to be EOL.

mosesn commented 3 years ago

Rad! I'm not going to try to push moving Twitter to 3.2 forward immediately then. I've taken a stab at Config, but it seems like we used it in a bad way with util-eval and we're going to need to unravel that.

smarter commented 3 years ago

I see there is a PR open now to bring 0.29 support to scalatest 3.1.x which I seemingly prematurely assumed to be EOL.

FWIW, it looks like scalatest 3.1 hasn't been published for dotty 0.27.0-RC1, so it might in fact be EOL by now: https://repo1.maven.org/maven2/org/scalatest/scalatest_0.27/, though it might be worth asking the maintainers.

mosesn commented 3 years ago

Ok, that's good to know. We'll start trying to push for scalatest 3.2. Scalafix seems to have a hard time on codebases of our size, so we're going to need to do some work to get us there.

martijnhoekstra commented 3 years ago

Scalafix seems to have a hard time on codebases of our size

Bug reports about that are probably appreciated, even if they don't turn out to be actionable in a timeframe to be useful to this issue.

mosesn commented 3 years ago

@martijnhoekstra we've basically built a distributed scalafix-ing framework that lets us chunk up scalafixing. I'm not sure there's a ton that scalafix can do to improve this–we just can't hold all of Twitter's scala code in RAM.

mosesn commented 3 years ago

OK, @joybestourous removed Config, and it looks like scalatest 3.2.x is published for scala 3, so I think the last thing we're waiting on is migrating to the newer scalatest now. We're still having some problems with the scalafix rule. Although the distributed scalafix job seems to be working, we're now having a problem with renaming collisions.

felixbr commented 3 years ago

FYI: Scalatest has support for all Scala 3 releases so far:

https://mvnrepository.com/artifact/org.scalatest/scalatest

Just thought it might be good to know 🙂

mosesn commented 3 years ago

Thanks for following up @felixbr! We have had some issues with the scalatest 3.2.x migration (short version is that scalafix runs into a ton of edge cases it doesn't handle well in the rest of our codebase) and I haven't had a chance yet to either fix the edge cases by hand or patch scalafix. I think if someone wants to make forward progress, they should try building util against 3.1.x in Scala 2.x and 3.2.x in Scala 3.x so that we don't get blocked by this.

felixbr commented 3 years ago

I've opened PRs #288 and #289 as preparation steps for a real Scala 3.0.0 cross-build. 🙂

felixbr commented 2 years ago

It's been a while since I've last looked into it and a lot of util-* modules support Scala 3 now, which is great!

The end goal for me personally is to use Finagle and twitter-server with Scala 3.

I checked the finagle project and most of the util-* modules finagle-core depends on are already ported. The ones missing seem to be:

Furthermore, scrooge (which is required for finagle-thrift) needs the following, which are also not ported yet:

I don't know if any of those aren't done because there are blockers or if there just wasn't any need so far. If you have any information about that, please let me know. 🙂

mosesn commented 2 years ago

@felixbr thanks for reaching out! util-validator is a new project, it was just an oversight to not publish it for 3.x. I'm not sure about util-{security,tunable,test}. We would be excited about PRs to get them publishable w/ 3.x!

pjfanning commented 2 years ago

util-jackson does not yet seem to support Scala 3

https://mvnrepository.com/artifact/com.twitter/util-jackson

jackson-module-scala supports Scala 3 but there is one issue that will affect you. jackson-module-scala ScalaObjectMapper is deprecated but is still supported on Scala 2 (and will be for a while) but for Scala 3, you will need to use ClassTagExtensions instead. ClassTagExtensions works in Scala 2 but there may be edge cases where jackson-module-scala ScalaObjectMapper works and ClassTagExtensions does not.

This is the util class that will need to change if you want to support Scala 3:

https://github.com/twitter/util/blob/develop/util-jackson/src/main/scala/com/twitter/util/jackson/ScalaObjectMapper.scala

pjfanning commented 2 years ago

Looks like a lot of modules depend on util-reflect which won't compile in Scala 3 because many scala reflection features have been replaced in Scala 3.

[info] compiling 3 Scala sources to /Users/pj.fanning/code/util/util-reflect/target/scala-3.0.2/classes ...
[error] -- Error: /Users/pj.fanning/code/util/util-reflect/src/main/scala/com/twitter/util/reflect/Classes.scala:15:65 
[error] 15 |  val simpleName: Class[_] => String = Memoize { clazz: Class[_] =>
[error]    |                                                                 ^
[error]    |parentheses are required around the parameter of a lambda
[error]    |This construct can be rewritten automatically under -rewrite -source 3.0-migration.
[error] -- Error: /Users/pj.fanning/code/util/util-reflect/src/main/scala/com/twitter/util/reflect/Types.scala:21:67 
[error] 21 |  val isCaseClass: Class[_] => Boolean = Memoize { clazz: Class[_] =>
[error]    |                                                                   ^
[error]    |parentheses are required around the parameter of a lambda
[error]    |This construct can be rewritten automatically under -rewrite -source 3.0-migration.
[error] -- [E008] Not Found Error: /Users/pj.fanning/code/util/util-reflect/src/main/scala/com/twitter/util/reflect/Types.scala:5:21 
[error] 5 |import scala.reflect.api.TypeCreator
[error]   |       ^^^^^^^^^^^^^^^^^
[error]   |       value api is not a member of reflect
[error] -- [E008] Not Found Error: /Users/pj.fanning/code/util/util-reflect/src/main/scala/com/twitter/util/reflect/Types.scala:6:21 
[error] 6 |import scala.reflect.runtime.universe._
[error]   |       ^^^^^^^^^^^^^^^^^^^^^
[error]   |       value runtime is not a member of reflect
[error] four errors found

Also util-mock depends on mockito-scala and that lib does not support Scala 3 - see https://github.com/mockito/mockito-scala/issues/364

felixbr commented 2 years ago

@mosesn There are several PRs for Scala 3 related updates open. Is there any chance to get them in?

I'd love to have everything in place so Scala 3 work on finagle-core is unblocked 🙂

pjfanning commented 1 year ago

The Scala team have clarified how best to support Scala 3.x builds.

I was under the false impression that building with Scala 3.0 was best but the article above suggests building with latest Scala version even if that prevents lib users from using the latest lib releases with older versions of Scala 3 libs.

Scala 3.2.0 was released recently.

felixbr commented 1 year ago

Hi, it's me again 👋

I've rebased and updated all my PRs regarding Scala 3 support (util-test, util-tunable, Adding Scala 3 to CI, and hopefully a fix for the flaky test in util-app).

I've tried to keep them as minimal as possible because I understand that Scala 3 might not be a high priority at Twitter currently and justifying time for that might be difficult.

That said, I'd appreciate if someone could merge the PRs or at least tell me what needs to be improved so they can be merged.

Cheers