Inconsistent test detection with project matrices

[keynmol]

Describe the bug

I'm testing the Test Explorer new feature which I'm super excited about.

There are two cases I've seen so far:

Only the first project gets picked up:

By first I mean "first defined in the matrix" - if I compile another file, I will get, say, openapi2_12 appearing - but not js3 or openapi3

Reproduction:

1. Clone https://github.com/disneystreaming/smithy4s/
2. Import build (it will take a long time)
3. Compile cascade
4. Wait for tests to appear

All projects are picked up but tests are not runnable

perhaps it's down to framework detection in SN

1. Clone https://github.com/indoorvivants/scala-library-template/pull/1 (it just has updates)
2. Import build
3. Compile cascade
4. Wait for tests to appear
5. Try to run tests from the view
6. Nothing will happen

Metals will output errors in the logs initially, but even JVM tests are not running:

2022.01.13 12:45:14 INFO  time: compiled core2_12 in 0.57s
2022.01.13 12:45:14 ERROR Referring to non-existent class org.scalajs.testing.bridge.Bridge
2022.01.13 12:45:14 ERROR   called from core module module initializers
2022.01.13 12:45:14 ERROR Referring to non-existent method static org.scalajs.testing.bridge.Bridge.start()void
2022.01.13 12:45:14 ERROR   called from core module module initializers
2022.01.13 12:45:14 ERROR JavaScript linking failed with 'There were linking errors'
2022.01.13 12:45:14 INFO  time: compiled core3 in 0.6s
2022.01.13 12:45:14 INFO  time: compiled coreJS3 in 0.62s
2022.01.13 12:45:15 INFO  time: compiled coreNative2_12 in 1.04s
2022.01.13 12:45:16 INFO  time: compiled core in 2.48s
2022.01.13 12:45:16 INFO  time: compiled coreJS in 2.82s
2022.01.13 12:45:16 INFO  time: compiled coreNative in 2.94s
2022.01.13 12:45:16 ERROR Detecting test frameworks in Scala Native projects is not yet supported
2022.01.13 12:46:13 INFO  compiling corenative2_12-test (1 scala source)
2022.01.13 12:46:14 INFO  time: compiled coreNative2_12-test in 1.24s
2022.01.13 12:46:14 ERROR Detecting test frameworks in Scala Native projects is not yet supported

Expected behavior

Tests are discovered from all the projects and are runnable.

Operating system

macOS

Editor/Extension

VS Code

Version of Metals

0.11.0+6-50088c55-SNAPSHOT

Extra context or search terms

test, explorer, vs code, mac os

[kpodsiad]

I took a quick look at this and it seems to be more than just a test explorer's issue. Saving changes on one of the files in test folder triggers compilation only at the core2_12-test target which will trigger rebuild index with only one build target identifier and test classes won't be found for the rest of the build targets (yeah, it's different behavior than reported in issue, but root cause is probably the same)

compiling core2_12-test (1 scala source)
time: compiled core2_12-test in 1.87s
compiling core2_12-test (1 scala source)
time: compiled core2_12-test in 1.48s

[tgodzik]

Thanks for reporting!

I think there are two issues here:

• we should run all targets initially and we doing cascade compile to update all tests. (related discussion https://github.com/scalameta/metals/discussions/3537)
• we should not suggest Native and JS tests as we cannot run them currently.

[ekrich]

Was wondering about a portion of this one. In Scala native projects we get the following:

ERROR Detecting test frameworks in Scala Native projects is not yet supported

Is this true and if so is there anything Scala Native can do to make this better?

[tgodzik]

Currently DAP is not supported for Scala Native and Scala JS, but we should actually be able to run it outside DAP. It would be cool to make it work since DAP setup for Java is talking a bit of time.