search

scalameta / metals

Scala language server with rich IDE features 🚀
https://scalameta.org/metals/
Apache License 2.0
2.04k stars 318 forks source link

Metals compile missing some projects in VS Code #5996

Open wecharyu opened 6 months ago

wecharyu commented 6 months ago

Describe the bug

Description I try to import Apache Spark project by Bloop and Metals in VS Code, but the metals will only compile some of the projects. Here is the part of Metals outputs:

Java home: /usr/lib/jvm/temurin-8-jdk-amd64
Metals version: 1.2.0
2024.01.04 14:06:47 INFO  logging to files /root/workspace/spark/.metals/metals.log
2024.01.04 14:06:47 INFO  Started: Metals version 1.2.0 in folders '/root/workspace/spark' for client Visual Studio Code 1.85.1.
Initializing Scala Debugger
Release notes was not shown: not showing release notes since they've already been seen for your current version
2024.01.04 14:06:48 WARN  Build server is not auto-connectable.
2024.01.04 14:06:48 WARN  no build target for: /root/workspace/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
2024.01.04 14:06:50 INFO  no build target found for /root/workspace/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala. Using presentation compiler with project's scala-library version: 3.3.1
2024.01.04 14:06:52 INFO  time: code lens generation in 4.28s
2024.01.04 14:07:44 WARN  no build target for: /root/workspace/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
2024.01.04 14:09:49 INFO  Attempting to connect to the build server...
2024.01.04 14:09:49 INFO  Bloop uses /usr/lib/jvm/java-17-openjdk-amd64 defined at /root/.bloop/bloop.json
2024.01.04 14:09:52 INFO  tracing is disabled for protocol BSP, to enable tracing of incoming and outgoing JSON messages create an empty file at /root/workspace/spark/.metals/bsp.trace.json or /root/.cache/metals/bsp.trace.json
2024.01.04 14:09:52 INFO  time: Connected to build server in 3.31s
2024.01.04 14:09:52 INFO  Connected to Build server: Bloop v1.5.13
2024.01.04 14:09:53 INFO  time: Imported build in 0.76s
2024.01.04 14:10:06 INFO  time: indexed workspace in 12s
2024.01.04 14:10:06 INFO  compiling spark-tags_2.12 (2 scala sources and 8 java sources)
2024.01.04 14:10:10 INFO  compiling spark-tags_2.12-test (6 java sources)
2024.01.04 14:10:10 INFO  time: compiled spark-tags_2.12 in 3.86s
2024.01.04 14:10:10 INFO  time: compiled spark-tags_2.12-test in 0.84s
2024.01.04 14:10:10 INFO  compiling spark-sketch_2.12 (9 java sources)
2024.01.04 14:10:10 INFO  compiling spark-kvstore_2.12 (12 java sources)
2024.01.04 14:10:10 INFO  compiling spark-launcher_2.12 (20 java sources)
2024.01.04 14:10:10 INFO  compiling spark-unsafe_2.12 (18 java sources)
2024.01.04 14:10:10 INFO  compiling spark-network-common_2.12 (84 java sources)
2024.01.04 14:10:12 INFO  time: compiled spark-sketch_2.12 in 1.22s
2024.01.04 14:10:12 INFO  time: compiled spark-unsafe_2.12 in 1.9s
2024.01.04 14:10:12 INFO  time: compiled spark-kvstore_2.12 in 1.9s
2024.01.04 14:10:12 INFO  time: compiled spark-launcher_2.12 in 1.9s
2024.01.04 14:10:14 INFO  compiling spark-network-common_2.12-test (26 java sources)
2024.01.04 14:10:14 INFO  time: compiled spark-network-common_2.12 in 2.62s
2024.01.04 14:10:16 INFO  compiling spark-network-shuffle_2.12 (50 java sources)
2024.01.04 14:10:16 INFO  time: compiled spark-network-common_2.12-test in 2.09s
2024.01.04 14:10:18 INFO  time: compiled spark-network-shuffle_2.12 in 2.04s
2024.01.04 14:10:18 INFO  compiling spark-network-yarn_2.12 (3 java sources)
2024.01.04 14:10:18 INFO  compiling spark-core_2.12 (567 scala sources and 104 java sources)
2024.01.04 14:10:19 INFO  time: compiled spark-network-yarn_2.12 in 1.39s
2024.01.04 14:10:34 INFO  time: compiled spark-core_2.12 in 16s

From the output result, metals only compile for spark-tags_2.12, spark-sketch_2.12, spark-kvstore_2.12 etc, but my projects number is 56, which means almost half projects would not be compiled.

Here is an example, spark-core_2.12 project was compiled and it could be attached in PACKAGES, while spark-core_2.12-test was not compiled, and we can not attach it in PACKAGES: image

Reproduction

  1. Export the build by mvn 
    mvn ch.epfl.scala:bloop-maven-plugin:2.0.0:bloopInstall -DdownloadSources=true -Pyarn -Phive -Phive-thriftserver
  2. Metals: Connect to build server

Expected behavior

Metals should compile all projects exported in .blood:

$ ls -larth .bloop 
total 7.6M
drwxr-xr-x 40 root root 4.0K Jan  4 14:08 ..
drwxr-xr-x  2 root root 4.0K Jan  4 14:09 .
-rw-r--r--  1 root root  431 Jan  4 14:09 bloop.settings.json
-rw-r--r--  1 root root  60K Jan  4 16:17 spark-parent_2.12.json
-rw-r--r--  1 root root  67K Jan  4 16:17 spark-parent_2.12-test.json
-rw-r--r--  1 root root  60K Jan  4 16:17 spark-tags_2.12.json
-rw-r--r--  1 root root  67K Jan  4 16:17 spark-tags_2.12-test.json
-rw-r--r--  1 root root  60K Jan  4 16:17 spark-sketch_2.12.json
-rw-r--r--  1 root root  67K Jan  4 16:17 spark-sketch_2.12-test.json
-rw-r--r--  1 root root  67K Jan  4 16:17 spark-kvstore_2.12.json
-rw-r--r--  1 root root  75K Jan  4 16:17 spark-kvstore_2.12-test.json
-rw-r--r--  1 root root  74K Jan  4 16:17 spark-network-common_2.12.json
-rw-r--r--  1 root root  81K Jan  4 16:17 spark-network-common_2.12-test.json
-rw-r--r--  1 root root  73K Jan  4 16:17 spark-network-shuffle_2.12.json
-rw-r--r--  1 root root  81K Jan  4 16:17 spark-network-shuffle_2.12-test.json
-rw-r--r--  1 root root  66K Jan  4 16:17 spark-unsafe_2.12.json
-rw-r--r--  1 root root  73K Jan  4 16:17 spark-unsafe_2.12-test.json
-rw-r--r--  1 root root  66K Jan  4 16:17 spark-launcher_2.12.json
-rw-r--r--  1 root root  74K Jan  4 16:17 spark-launcher_2.12-test.json
-rw-r--r--  1 root root 162K Jan  4 16:17 spark-core_2.12.json
-rw-r--r--  1 root root 173K Jan  4 16:17 spark-core_2.12-test.json
-rw-r--r--  1 root root 145K Jan  4 16:17 spark-mllib-local_2.12.json
-rw-r--r--  1 root root 157K Jan  4 16:17 spark-mllib-local_2.12-test.json
-rw-r--r--  1 root root 151K Jan  4 16:17 spark-graphx_2.12.json
-rw-r--r--  1 root root 157K Jan  4 16:17 spark-graphx_2.12-test.json
-rw-r--r--  1 root root 137K Jan  4 16:17 spark-streaming_2.12.json
-rw-r--r--  1 root root 144K Jan  4 16:17 spark-streaming_2.12-test.json
-rw-r--r--  1 root root 146K Jan  4 16:17 spark-catalyst_2.12.json
-rw-r--r--  1 root root 153K Jan  4 16:17 spark-catalyst_2.12-test.json
-rw-r--r--  1 root root 168K Jan  4 16:17 spark-sql_2.12.json
-rw-r--r--  1 root root 177K Jan  4 16:17 spark-sql_2.12-test.json
-rw-r--r--  1 root root 177K Jan  4 16:17 spark-mllib_2.12.json
-rw-r--r--  1 root root 185K Jan  4 16:17 spark-mllib_2.12-test.json
-rw-r--r--  1 root root  67K Jan  4 16:17 spark-tools_2.12.json
-rw-r--r--  1 root root  74K Jan  4 16:17 spark-tools_2.12-test.json
-rw-r--r--  1 root root 192K Jan  4 16:17 spark-hive_2.12.json
-rw-r--r--  1 root root 200K Jan  4 16:17 spark-hive_2.12-test.json
-rw-r--r--  1 root root 176K Jan  4 16:17 spark-repl_2.12.json
-rw-r--r--  1 root root 186K Jan  4 16:17 spark-repl_2.12-test.json
-rw-r--r--  1 root root  76K Jan  4 16:17 spark-network-yarn_2.12.json
-rw-r--r--  1 root root  83K Jan  4 16:17 spark-network-yarn_2.12-test.json
-rw-r--r--  1 root root 172K Jan  4 16:17 spark-yarn_2.12.json
-rw-r--r--  1 root root 181K Jan  4 16:17 spark-yarn_2.12-test.json
-rw-r--r--  1 root root 195K Jan  4 16:17 spark-hive-thriftserver_2.12.json
-rw-r--r--  1 root root 202K Jan  4 16:17 spark-hive-thriftserver_2.12-test.json
-rw-r--r--  1 root root 220K Jan  4 16:17 spark-assembly_2.12.json
-rw-r--r--  1 root root 228K Jan  4 16:17 spark-assembly_2.12-test.json
-rw-r--r--  1 root root 138K Jan  4 16:17 spark-token-provider-kafka-0-10_2.12.json
-rw-r--r--  1 root root 145K Jan  4 16:17 spark-token-provider-kafka-0-10_2.12-test.json
-rw-r--r--  1 root root 154K Jan  4 16:17 spark-streaming-kafka-0-10_2.12.json
-rw-r--r--  1 root root 164K Jan  4 16:17 spark-streaming-kafka-0-10_2.12-test.json
-rw-r--r--  1 root root 190K Jan  4 16:17 spark-sql-kafka-0-10_2.12.json
-rw-r--r--  1 root root 202K Jan  4 16:17 spark-sql-kafka-0-10_2.12-test.json
-rw-r--r--  1 root root 212K Jan  4 16:17 spark-examples_2.12.json
-rw-r--r--  1 root root 219K Jan  4 16:17 spark-examples_2.12-test.json
-rw-r--r--  1 root root 138K Jan  4 16:17 spark-streaming-kafka-0-10-assembly_2.12.json
-rw-r--r--  1 root root 145K Jan  4 16:17 spark-streaming-kafka-0-10-assembly_2.12-test.json
-rw-r--r--  1 root root 158K Jan  4 16:17 spark-avro_2.12.json
-rw-r--r--  1 root root 166K Jan  4 16:17 spark-avro_2.12-test.json

Operating system

Linux

Editor/Extension

VS Code

Version of Metals

v1.2.0

Extra context or search terms

Bloop version: v1.5.13 Scala version: v2.12.18 Zinc version: v1.9.5 Metals version: v1.2.0 JDK: 1.8

tgodzik commented 6 months ago

Thanks for reporting! By default we usually compile only the current and dependent projects based on the current open files. That usually makes most of the files compile. Was there any issues with compilation by any chance? That would stop downstream projects from being compiled.

wecharyu commented 6 months ago

There was not compile error message in the log, so it's a bit strange. BTW, I have also tried with Spark master code, and it can compile all the projects. I'm not sure why it does not work for my internal Spark code (v3.2.1) even though they share the same configurations.

tgodzik commented 6 months ago

Hard to say, do you have a link to a problematic tag/branch?

wecharyu commented 6 months ago

@tgodzik Can try with Spark v3.2.0.