FetchError on Travis CI

[hseeberger]

• Version: 0.6.2
• Integration: sbt

On Travis CI I get the following errors (5 times) for a project with 6 sub-projects:

org.scalafmt.bootstrap.FetchError: ArrayBuffer((Dependency(com.geirsson:scalafmt-cli_2.11,0.6.2,compile,Set(),Attributes(jar,),false,true),List(not found: /home/travis/.ivy2/local/com.geirsson/scalafmt-cli_2.11/0.6.2/ivys/ivy.xml, locked: /home/travis/.coursier/cache/v1/https/repo1.maven.org/maven2/com/geirsson/scalafmt-cli_2.11/0.6.2/scalafmt-cli_2.11-0.6.2.pom)))
    at org.scalafmt.bootstrap.ScalafmtBootstrap$.fromVersionUncached(Bootstrap.scala:71)
    at org.scalafmt.bootstrap.ScalafmtBootstrap$$anonfun$fromVersion$1.apply(Bootstrap.scala:45)
    at org.scalafmt.bootstrap.ScalafmtBootstrap$$anonfun$fromVersion$1.apply(Bootstrap.scala:45)
    at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
    at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
    at org.scalafmt.bootstrap.ScalafmtBootstrap$.fromVersion(Bootstrap.scala:45)
    at org.scalafmt.bootstrap.ScalafmtBootstrap$.main(Bootstrap.scala:94)
    at AutomateScalafmtPlugin$autoImport$$anonfun$automateScalafmtFor$1$$anonfun$apply$3.AutomateScalafmtPlugin$autoImport$$anonfun$$anonfun$$formattingHandler$1(AutomateScalafmtPlugin.scala:48)

On my local machine everything is fine.

[olafurpg]

Are you using coursier for the build? What happens if you run "sbt scalafmt --help" before compile? On Thu, 9 Mar 2017 at 17:24, Heiko Seeberger notifications@github.com wrote:

• Version: 0.6.2
• Integration: sbt

On Travis CI I get the following errors (5 times) for a project with 6 sub-projects:

org.scalafmt.bootstrap.FetchError: ArrayBuffer((Dependency(com.geirsson:scalafmt-cli_2.11,0.6.2,compile,Set(),Attributes(jar,),false,true),List(not found: /home/travis/.ivy2/local/com.geirsson/scalafmt-cli_2.11/0.6.2/ivys/ivy.xml, locked: /home/travis/.coursier/cache/v1/https/repo1.maven.org/maven2/com/geirsson/scalafmt-cli_2.11/0.6.2/scalafmt-cli_2.11-0.6.2.pom))) at org.scalafmt.bootstrap.ScalafmtBootstrap$.fromVersionUncached(Bootstrap.scala:71) at org.scalafmt.bootstrap.ScalafmtBootstrap$$anonfun$fromVersion$1.apply(Bootstrap.scala:45) at org.scalafmt.bootstrap.ScalafmtBootstrap$$anonfun$fromVersion$1.apply(Bootstrap.scala:45) at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189) at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91) at org.scalafmt.bootstrap.ScalafmtBootstrap$.fromVersion(Bootstrap.scala:45) at org.scalafmt.bootstrap.ScalafmtBootstrap$.main(Bootstrap.scala:94) at AutomateScalafmtPlugin$autoImport$$anonfun$automateScalafmtFor$1$$anonfun$apply$3.AutomateScalafmtPlugin$autoImport$$anonfun$$anonfun$$formattingHandler$1(AutomateScalafmtPlugin.scala:48)

On my local machine everything is fine.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/olafurpg/scalafmt/issues/807, or mute the thread https://github.com/notifications/unsubscribe-auth/ABV8XZnWgQw3OJdZmW8DjOV6y-tyv9wDks5rkCeugaJpZM4MYTtr .

[hseeberger]

We are talking about this project: https://github.com/hseeberger/akka-http-json. The issue only occurs on Travis CI, not locally.

I don't use coursier, neither on my laptop nor on Travis CI.

I can run scalafmt --help locally in an sbt session which produces the following output, but I can't do that on Travis CI:

~/projekte/hseeberger/akka-http-json(master ✔) sbt
[info] Loading global plugins from /Users/heiko/.sbt/0.13/plugins
[info] Loading project definition from /Users/heiko/projekte/hseeberger/akka-http-json/project
[info] Set current project to akka-http-json (in build file:/Users/heiko/projekte/hseeberger/akka-http-json/)
[akka-http-json]> scalafmt --help
scalafmt 0.6.2
Usage: scalafmt [options]

  -h, --help               prints this usage text
  -v, --version            print version
  -f, --files <value>      file or directory, in which case all *.scala files are formatted.
  --exclude <value>        file or directory, in which case all *.scala files are formatted.
  -c, --config <value>     a file path to .scalafmt.conf.
  --config-str <value>     configuration defined as a string
  --stdin                  read from stdin and print to stdout
  --assume-filename <value>
                           required to format .sbt files with --stdin flag.
  -i, --in-place           write output to file, does nothing if file is not specified
  --test                   test for mis-formatted code, exits with status 1 on failure.
  --migrate2hocon <value>  migrate .scalafmt CLI style configuration to hocon style configuration in .scalafmt.conf
  --diff                   If set, only format edited files in git diff against master.
  --diff-branch <value>    If set, only format edited files in git diff against provided branch.
  --build-info             prints build information
  --quiet                  don't print out stuff to console.
  --debug                  print out diagnostics to console.
  --non-interactive        disable fancy progress bar, useful in ci or sbt plugin.
Examples:
scalafmt # Format all files in the current project, configuration is determined in this order:
         # 1. .scalafmt.conf file in current directory
         # 2. .scalafmt.conf inside root directory of current git repo
         # 3. no configuration, default style
scalafmt --test # throw exception on mis-formatted files, won't write to files.
scalafmt --diff # Format all files that were edited in git diff against master branch.
scalafmt --diff-branch 2.x # same as --diff, except against branch 2.x
scalafmt --stdin # read from stdin and print to stdout
scalafmt --stdin --assume-filename foo.sbt # required to format .sbt files
scalafmt -f Code.scala             # print formatted contents to stdout.
scalafmt -i -f Code1.scala,A.scala # write formatted contents to file.
scalafmt -i -f . --exclude target  # format all files in directory excluding target
scalafmt --config .scalafmt.conf   # read custom style from file
scalafmt --config-str "style=IntelliJ" # define custom style as a flag, must be quoted.
Please file bugs to https://github.com/olafurpg/scalafmt/issues

[olafurpg]

I'm unable to investigate this right now, since I'm out of town skiing. It smells of a race condition between the ivy resolver and coursier.

The scalafmt sbt plugin uses the coursier api, I suspect the bug is in the scalafmt bootstrap project.

As a temporary workaround, you can try the coursier bootstrap script as documented on the website. We use that for CI in the scalameta repo with great success (Scala native does the same). On Thu, 9 Mar 2017 at 22:41, Heiko Seeberger notifications@github.com wrote:

We are talking about this project: https://github.com/hseeberger/akka-http-json. The issue only occurs on Travis CI, not locally.

I don't use coursier, neither on my laptop nor on Travis CI.

I can run scalafmt --help locally in an sbt session which produces the following output, but I can't do that on Travis CI:

~/projekte/hseeberger/akka-http-json(master ✔) sbt [info] Loading global plugins from /Users/heiko/.sbt/0.13/plugins [info] Loading project definition from /Users/heiko/projekte/hseeberger/akka-http-json/project [info] Set current project to akka-http-json (in build file:/Users/heiko/projekte/hseeberger/akka-http-json/) [akka-http-json]> scalafmt --help scalafmt 0.6.2 Usage: scalafmt [options]

-h, --help prints this usage text -v, --version print version -f, --files file or directory, in which case all .scala files are formatted. --exclude file or directory, in which case all .scala files are formatted. -c, --config a file path to .scalafmt.conf. --config-str configuration defined as a string --stdin read from stdin and print to stdout --assume-filename required to format .sbt files with --stdin flag. -i, --in-place write output to file, does nothing if file is not specified --test test for mis-formatted code, exits with status 1 on failure. --migrate2hocon migrate .scalafmt CLI style configuration to hocon style configuration in .scalafmt.conf --diff If set, only format edited files in git diff against master. --diff-branch If set, only format edited files in git diff against provided branch. --build-info prints build information --quiet don't print out stuff to console. --debug print out diagnostics to console. --non-interactive disable fancy progress bar, useful in ci or sbt plugin. Examples: scalafmt # Format all files in the current project, configuration is determined in this order:

1. .scalafmt.conf file in current directory

     # 2. .scalafmt.conf inside root directory of current git repo
     # 3. no configuration, default style

scalafmt --test # throw exception on mis-formatted files, won't write to files. scalafmt --diff # Format all files that were edited in git diff against master branch. scalafmt --diff-branch 2.x # same as --diff, except against branch 2.x scalafmt --stdin # read from stdin and print to stdout scalafmt --stdin --assume-filename foo.sbt # required to format .sbt files scalafmt -f Code.scala # print formatted contents to stdout. scalafmt -i -f Code1.scala,A.scala # write formatted contents to file. scalafmt -i -f . --exclude target # format all files in directory excluding target scalafmt --config .scalafmt.conf # read custom style from file scalafmt --config-str "style=IntelliJ" # define custom style as a flag, must be quoted. Please file bugs to https://github.com/olafurpg/scalafmt/issues

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/olafurpg/scalafmt/issues/807#issuecomment-285492429, or mute the thread https://github.com/notifications/unsubscribe-auth/ABV8XWSrDoO-9gTilgTeiVN38YqEHo0_ks5rkHIOgaJpZM4MYTtr .

[hseeberger]

Not sure how the bootstrap script would help with the CI server issue. But hey, maybe I could switch to sbt-coursier ...

Enjoy skiing.

[olafurpg]

In scala.meta, we put a 16kb scalafmt script generated by coursier here: https://github.com/scalameta/scalameta/blob/061c6226dbe5c9c0d948c1ddef190973816b4622/bin/scalafmt and run that script in CI like this https://github.com/scalameta/scalameta/blob/061c6226dbe5c9c0d948c1ddef190973816b4622/.drone.yml#L20

To generate the bootstrap script, we use the command here: https://olafurpg.github.io/scalafmt/#Coursier

The script downloads and runs on the entire repo in ~4s before we start sbt.

This issue is hard to fix since I'm unable to reproduce. I have an idea of what might be causing it, see https://github.com/olafurpg/scalafmt/pull/817 We'll probably have to wait until the next release to test if the fix works as I hope it does.

[hseeberger]

Thanks for the explanation. So you're saying that by running the script before running sbt, all needed downloads are already in place? Sounds cool, I'll give it a try!

[olafurpg]

The bootstrap script puts the dependencies in ~/.coursier/bootstrap while sbt-coursier/sbt-scalafmt use ~/.coursier/cache (I don't know why it's different). But if you run coursier fetch com.geirsson:scalafmt-cli_2.11:0.6.2 (see https://github.com/alexarchambault/coursier#command-line), the downloads will be in place in ~/.coursier/cache.

[hseeberger]

I still get the same error :-(

Here's what I did:

• Created a bootstrap script in the root directory of my project:
  coursier bootstrap com.geirsson:scalafmt-cli_2.11:0.6.2 --main org.scalafmt.cli.Cli -o scalafmt
• Changed .travis.yml to execute that script before the actual test script is run:
  before_script: $TRAVIS_BUILD_DIR/scalafmt

Looking at the build output the new scalafmt script was executed without errors. But it didn't show any effect.

Just to be sure: should that help or should I additionally use the sbt-coursier plugin?

[olafurpg]

The bootstrap script won't help the sbt-scalafmt, since they use a different cache. Running the bootstrap script is the same as running sbt scalafmt, they both call the cli. Using sbt-coursier probably won't help either, the issue seems be related to a race in ~/.ivy2/local which the next version of sbt-scalafmt will not use anymore.

[hseeberger]

OK, but this was a nice exercise anyway ;-)

Looking forward to the next release!

[olafurpg]

I just released 0.6.3, please let me know if the issue has now been resolved :)

[hseeberger]

Thanks! The issue has not been resolved. Now I can even reproduce it on my local machine! Please go to https://github.com/hseeberger/akka-http-json/tree/upgrades, enter sbt, then run test. You might have to rename your local ~/.coursier directory or at least delete the dependencies which are fetched for this project.

[olafurpg]

I am able to reproduce on akka-http-json :( It seems to be caused by the fact that compile resolution is conflicting with scalafmt resolution, now it's in the .coursier directory. I guess we can solve this by using another cache directory than ~/.coursier.

PS. This would not be a problem if sbt used a modern Scala version instead of 2.10 (https://twitter.com/fommil/status/841320967846690816)

[hseeberger]

I just changed akka-http-json to no longer use sbt-coursier, but the error is still the same: https://github.com/hseeberger/akka-http-json/pull/133

[olafurpg]

I just did a quick 0.6.4 release with https://github.com/scalameta/scalafmt/pull/834/files#diff-52cd850066d5f615d60e059df63deca0R70 It changes the defult cache directory to ~/.cache/sbt-scalafmt instead of ~/.coursier. You can override it with the SCALAFMT_CACHE environment variable. Please let me know if that fixes the issue.

PS. Please don't be discouraged by sbt-coursier because of this issue, sbt-coursier is such a productivity booster and the fault is 100% on the sbt-scalafmt side.

[stephennancekivell]

Hi guys, I think I've been investigating the same issue. At https://github.com/akka/alpakka/pull/216.

I've worked around it by making sure the sbt auto format plugin has a single reference to ScalafmtBootstrap. I think sbt is calling ScalafmtBootstrap.main in parallel then missing internal cache ScalafmtBootstrap.cliCache.

https://github.com/akka/alpakka/pull/216/commits/15228beb0cc9f9b9abf5afb8ec85061ece2a11d2

I think a better solution would be if coursier could be called with a lock timeout or something. It seems to just throw an exception straight away. When we do ... val resolution = start.process.run(fetch).unsafePerformSync

Heres some useful discussion of the issue https://github.com/alexarchambault/coursier/issues/411

[hseeberger]

@olafurpg same issue with 0.6.4

@stephennancekivell that makes total sense to me, I'll give it a try

[stephennancekivell]

@hseeberger, I've added another possible solution here. https://github.com/scalameta/scalafmt/pull/838

[hseeberger]

@olafurpg I think the reason is that you have used a plain mutable map for the cache: https://github.com/scalameta/scalafmt/blob/master/bootstrap/src/main/scala/org/scalafmt/bootstrap/Bootstrap.scala#L41. Could you try to use something thread-safe here?

[olafurpg]

Please reopen if this is still an issue.