scalajs-bundler web pipeline stage performance issues

[mdedetrich]

In one of our projects at work we have a case where run command takes ~4 minutes (this is even when the project is fully compiled, i.e. using run on the fully compiled project which skips fastOptJs/fullOptJs takes ~4 minutes).

A ticket for the project was made on the commercial lightbend support to try and diagnose what is causing sbt run to take so long, and the primary hit is scalajs-bundler web pipeline stage, i.e. https://github.com/scalacenter/scalajs-bundler/blob/master/sbt-web-scalajs-bundler/src/main/scala/scalajsbundler/sbtplugin/WebScalaJSBundlerPlugin.scala#L91 .

The interesting thing is that the project in question is using BundlingMode.LibraryOnly() and none of the npm dependencies are being changed so I am also questioning as to why even npm/bundler is even being run if none of the sources in the project haven't changed (the packaging should be basically instant).

For the reason above it also seems like there isn't any caching being done even though nothing has changed (in terms of sources).

Unfortunately since this is a commercial project I can't provide a reproducible case, however its available as a ticket in the Lightbend commercial support (you can contact me by email/gitter for more details)

[dispalt]

I am guessing you are using play correct? If so have you tried this? https://github.com/playframework/playframework/issues/5242#issuecomment-299562213

[MasseGuillaume]

@mdedetrich

This might help:

• https://github.com/scalacenter/scastie/tree/master/client/webpack
• https://github.com/scalacenter/scastie/blob/28a8b847a1121fe7c99c8177fdc526f64fcc0705/build.sbt#L312-L337

[mdedetrich]

@dispalt We don't use play so I don't think thats the issue. @MasseGuillaume Thanks I will have a look at your configs and try stuff out

[mdedetrich]

@MasseGuillaume So I had a look at your config and I noticed a few things. It seems that your project is completely based around using webpack/yarn for serving everything client side where as our project I have has a hybrid of the two, i.e. we have a lot of assets that are served by server (i.e. we use sbt-less to generate our assets rather than webpack).

The main speed improvement in your case seems to be from the usage of webpacks devServer + hot reloading for anything client related. This means we would have to restructure our project to get it to work purely with webpack (as well as seeing how to get webpacks devServer to work in conjunction with scala.js and our actual server api)

In any case there is an issue here, which is that if nothing changes, then sbt/sbt-scalajs-bundler should not be re-running anything at all, i.e. it shouldn't even be calling webpack.

[MasseGuillaume]

Take a look at: https://github.com/scalacenter/scastie/blob/master/project/ScalaJsHelper.scala#L10 https://github.com/scalacenter/scastie/blob/master/build.sbt#L274

[MasseGuillaume]

I had a call with @mdedetrich and we found the issue.

It boils down to the following:

server
  (managedClasspath in Runtime) += (packageBin in Assets).value 
  // << this wants the assets in fullOptJs
  // taken from: https://github.com/sbt/sbt-web#packaging-and-publishing
  // + sbt less

client (Lots of Scala.js)
    webpackBundlingMode in fastOptJS := BundlingMode.LibraryOnly, // fast
    webpackBundlingMode in fullOptJS := BundlingMode.Application,  // slow ~4min

I will create a reproduction repo and try to find a solution.

Additional info:

addSbtPlugin("ch.epfl.scala" % "sbt-web-scalajs-bundler" % "0.10.0")
addSbtPlugin("com.typesafe.sbt" % "sbt-less" % "1.1.2")

[MasseGuillaume]

https://github.com/MasseGuillaume/211

[julienrf]

@mdedetrich What’s the status of that issue?

[mdedetrich]

@MasseGuillaume @julienrf I tried replicating the changes in this commit https://github.com/MasseGuillaume/211/commit/75c9517571a4cbf508b6a512d4a18b9452e74a5e and the web assets were not being compiled.

I am not sure if I am missing something?

[julienrf]

What happens if you don’t remove that line?

(managedClasspath in Runtime) += (packageBin in Assets).value

[mdedetrich]

I am still experiencing the performance issue, run is still taking ~4 minutes.

Here are my current server plugins

  .enablePlugins(
    WebScalaJSBundlerPlugin,
    JavaAppPackaging
  )

and here are my client plugins

  .enablePlugins(ScalaJSBundlerPlugin, ScalaJSWeb)

I have also enabled (managedClasspath in Runtime) += (packageBin in Assets).value

@MasseGuillaume Do you need more help in reproducing this again?

[cquiroz]

@mdedetrich Have you tried with webpack4? I've seen great improvements in performance using version 4

[mdedetrich]

@cquiroz With @MasseGuillaume we managed to diagnose the issue but I couldn't get a fix working. The issue is due to the combination of sbt-assets and webpack, webpack was being run multiple times.

[fooblahblah]

I'm sure those that debugged this know it already, but if I run scalaJSPipeline from my Play server project it takes roughly ~5s to do nothing. That task is being triggered by any mods in the /public/ dir. If I just touch a .css it'll trigger scalaJSPipeline. If we could smartly detect there were no changes and return the previous bundle that might be a way to solve this.

[homebay] $ scalaJSPipeline
[debug] Copy resource mappings: 
[debug]     
[debug] [zinc] IncrementalCompile -----------
[debug] IncrementalCompile.incrementalCompile
[debug] previous = Stamps for: 94 products, 11 sources, 19 libraries
[debug] current source = Set(${BASE}/client/src/main/scala/admin/pages/property/LandingPage.scala, ${BASE}/client/src/main/scala/admin/pages/property/SMSLog.scala, ${BASE}/client/src/main/scala/admin/util/Debounce.scala, ${BASE}/client/src/main/scala/admin/pages/property/CallLog.scala, ${BASE}/client/src/main/scala/admin/pages/property/ShowingTimeRestrictionsApp.scala, ${BASE}/client/src/main/scala/admin/util/Format.scala, ${BASE}/client/src/main/scala/admin/util/Implicits.scala, ${BASE}/client/src/main/scala/admin/util/Styles.scala, ${BASE}/client/src/main/scala/admin/util/StyleBuilder.scala, ${BASE}/client/src/main/scala/admin/pages/property/AuditLog.scala, ${BASE}/client/src/main/scala/models/ShowingTimeRestrictions.scala)
[debug] > initialChanges = InitialChanges(Changes(added = Set(), removed = Set(), changed = Set(), unmodified = ...),Set(),Set(),API Changes: Set())
[debug] No changes
[debug] Created transactional ClassFileManager with tempDir = /home/jsimpson/workspace/homebay/client/target/scala-2.13/classes.bak
[debug] Removing the temporary directory used for backing up class files: /home/jsimpson/workspace/homebay/client/target/scala-2.13/classes.bak
[debug] Update IR cache: 14300 us
[success] Total time: 4 s, completed Apr 29, 2021, 10:42:25 PM