Getting `Failed to connect with build server....` while using devcontainer

[ndlarsen]

Describe the bug

Using a devcontainer, on the first fresh import, or any subsequent container rebuild, metals-vscode fails with Failed to connect with build server, no functionality will work. See logs for more details., Reloading the project, using the previously built container, works as expected. All subsequent reloads without a container rebuild works as well. This behaviour seems pretty consistent.

To be fair, I am not fully convinced this is an issue with nor caused by metals-vscode as I am using a devcontainer and a custom definition at that. Who knows, this might even come down to local timing issues. Anyway, here's a bugreport. Please let me know if any additional information is required.

To Reproduce Steps to reproduce the behaviour:

1. ensure all related containers not present
2. import/(re)open a project
3. observe devcontainer being built
4. observer metals-vscode fail with Failed to connect with build server, no functionality will work. See logs for more details.
5. reimport the project
6. observe metals-vscode succeed

Expected behaviour

metals-vscode succeeding when connecting

Installation:

• Operating system:

  $ lsb_release -a
  No LSB modules are available.
  Distributor ID:   Ubuntu
  Description:  Ubuntu 22.04.3 LTS
  Release:  22.04
  Codename: jammy

• VSCode version:

  Version: 1.82.2
  Commit: abd2f3db4bdb28f9e95536dfa84d8479f1eb312d
  Date: 2023-09-14T05:51:20.981Z
  Electron: 25.8.1
  ElectronBuildId: 23779380
  Chromium: 114.0.5735.289
  Node.js: 18.15.0
  V8: 11.4.183.29-electron.0
  OS: Linux x64 6.2.0-33-generic

• VSCode extension version: v1.25.0
• Metals version: 1.0.1

Additional context

• .devcontainer.json:

  {
  "runArgs": ["--name", "scala-devcontainer-poc"],
  "name": "scala-devcontainer-poc",
  "build": {
    "dockerfile": "./Dockerfile",
    "context": "."
  },
  "remoteUser": "vscode",
  "customizations": {
    "vscode": {
        "extensions": [
          "ms-azuretools.vscode-docker",
          "scala-lang.scala",
          "scalameta.metals",
        ],
    }
  },
  "forwardPorts": [
    5000,
    9000,
  ],
  }

• Dockerfile:

  ARG SOURCE_IMG_TAG=22.04
  ARG SOURCE_IMG_SHA256SUM=ec050c32e4a6085b423d36ecd025c0d3ff00c38ab93a3d71a460ff1c44fa6d77
  
  FROM ubuntu:${SOURCE_IMG_TAG}@sha256:${SOURCE_IMG_SHA256SUM}
  
  ARG COURSIER_VERSION=2.1.7
  ARG JAVA_DIST=temurin
  ARG JAVA_VERSION=1.11.0.20.1
  ARG AMMONITE_VERSION=1.9.6
  ARG SBT_VERSION=1.9.6
  ARG GITER8_VERSION=0.16.2
  ARG SCALAFMT_VERSION=3.7.14
  ARG SCALAFIX_VERSION=0.11.1
  ARG BIN_PATH=.local/bin
  
  ARG USERNAME=vscode 
  
  RUN groupadd ${USERNAME} \
    && useradd -rm -d /home/${USERNAME} -s /bin/bash -g ${USERNAME} ${USERNAME}
  
  RUN apt -q update && apt upgrade -qy && apt install -qy curl git
  
  USER ${USERNAME}
  WORKDIR /home/${USERNAME}
  
  RUN mkdir -p /home/${USERNAME}/${BIN_PATH}
  
  RUN curl -fL "https://github.com/coursier/coursier/releases/download/v${COURSIER_VERSION}/cs-x86_64-pc-linux.gz" | gzip -d > ${BIN_PATH}/cs \
    && chmod +x ${BIN_PATH}/cs
  
  RUN ${BIN_PATH}/cs java --jvm ${JAVA_DIST}:${JAVA_VERSION} --setup
  RUN ${BIN_PATH}/cs install --dir ~/${BIN_PATH} \
    ammonite:${AMMONITE_VERSION} \
    sbt:${SBT_VERSION} \
    giter8:${GITER8_VERSION}
    scalafmt:${SCALAFMT_VERSION}
    scalafix:${SCALAFIX_VERSION}

Logs

output.metals.log terminal.log

Project files

devcontainer-scala.zip

Search terms

devcontainer docker

[tgodzik]

Thanks for reporting! Is it possible that some files are not kept in the devcontainer?

2023.10.05 07:32:02 WARN  Scala instance jar /home/vscode/.sbt/boot/scala-2.12.18/lib/scala-library.jar doesn't exist!

This should exist locally since the build was exported with those paths and it looks like they were removed (not cached)

It's possible to change the directories for keeping sbt and coursier caches, some examples here https://github.com/scalameta/metals-gitpod-sample/blob/master/gitpod_init.sh

[ndlarsen]

Hi @tgodzik

I appreciate your reply.

Thanks for reporting! Is it possible that some files are not kept in the devcontainer?

Generally, no, the entire project directory is mounted into the container under /workspace.

2023.10.05 07:32:02 WARN  Scala instance jar /home/vscode/.sbt/boot/scala-2.12.18/lib/scala-library.jar doesn't exist!

This should exist locally since the build was exported with those paths and it looks like they were removed (not cached)

These are only warnings related to non-cached files, though. If the presence of any file in the cache is a hard requirement, and breaks "normal" application flow, then I would argue that the log level is incorrect and that it is a weird design choice to begin with as caches are ephemeral by nature and cannot be relied upon. Really, if a given file is not present in the cache, the application should fetch and populate but I digress...

It's possible to change the directories for keeping sbt and coursier caches, some examples here https://github.com/scalameta/metals-gitpod-sample/blob/master/gitpod_init.sh

The downside of manually populating the cache is that one would need to update the container definition on each version change of both Scala and Metals, which I generally would like to avoid. Anyway, I tried it out (without cached container layers) just to see if this would change anything and as expected it did not. Rolling back the change and rebuilding (again without cached container layers) for some reason succeeded this time, even though the warnings you noted above related to the non-cached files was still present.

So, it is after all not super consistent and clearly there is a bug somewhere but as previously stated I am currently unable to pinpoint the cause. This just strengthens my belief that this is caused by either metals-vscode, ms-vscode-remote.remote-containers, scalameta.metals, any of the underlying dependencies or an interaction between any of the aforementioned. It does, though, sound a lot like a race condition somewhere and also a lot like some the bloop related issues I have seen reports of as well.

~Given my inability to pin-point the cause and the lack of consistent reproducibility, feel free to close the issue. After all, I can reopen it or open a new one if new information comes to light.~

Thanks again. Regards.

[ndlarsen]

Okay, so, quick update. Still unpredictable behaviour but I recently noticed the following:

• I opened the project in vscode
• chose import build from the presented prompt
• after a while metals-vscode failed with Failed to connect with build server, no functionality will work. See logs for more details
• prompt presented with installing bloop
• rest of import/setup seems to complete okay

I have attached the relevant part of metals.log + server.log and extension.remote.log.

In addition to this, I have noticed that an import of exactly the same configuration with exactly the same project files from time to time fails with different errors, java.util.concurrent.ExecutionException: org.eclipse.lsp4j.jsonrpc.MessageIssueException: Message could not be parsed. being one of them.

I hope this information brings you closer to pin-pointing a possible cause.

metals.log extension.host.remote.log server.log

[tgodzik]

Looks like the issue is indeed related to using json files generated in previous sessions.

When you run build import, the sbt Bloop plugin will populate .bloop/*.json files with paths that Bloop can use (it doesn't actually fetch most dependencies, though that could be changed.). When dev container is turned off, those caches are lost, but the json files contain those paths. That's why I suggested using the same trick we have in the example repo. You don't actually need to prepopulate the repos there, but the additional coursier cache setting make sure that artifacts are kept between sessions.

The reason for all this is that Bloop is just a build server, so it doesn't deal with resolution. The workaround here is to always import build when starting a dev container or use sbt via BSP, which will automatically download everything each time. You can use Metals: Switch build server command for that. I would however recommend tinkering with caches as that can reduce the startup time.