ucb-bar / chipyard

An Agile RISC-V SoC Design Framework with in-order cores, out-of-order cores, accelerators, and more
https://chipyard.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
1.67k stars 657 forks source link

Issues using Metals (Scala Lanugage Server) with Chipyard in VSCode #986

Open tymcauley opened 3 years ago

tymcauley commented 3 years ago

Impact: other

Tell us about your environment: Chipyard Version: Current dev branch (64aa972) OS: macOS 11.6 Other:

$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Mac OS X amd64-64-Bit Compressed References 20210421_957 (JIT enabled, AOT enabled)
OpenJ9   - b4cc246d9
OMR      - 162e6f729
JCL      - 7796c80419 based on jdk-11.0.11+9)

What is the current behavior?

I'm trying to get Metals up and running for Chipyard, but running into a number of issues (can't find many definitions, lots of spurious errors in the workspace related to the chisel3 package not getting linked in properly, etc). Here is how I set it up:

Once Metals runs, you can find its log file at .metals/metals.log (located in the root of the chipyard repo). I have attached that file here: metals.log

Inside IOBinders.scala, if I right-click on IOBinderTuple on line 59, and choose "Go to Definition", it takes me there (line 30 of the same file). However, if I do the same for LazyModule on line 81, I get the message "No definition found for 'LazyModule'". In both cases, I see this message show up in metals.log:

2021.09.19 11:14:05 ERROR code navigation does not work for the file '/Users/tynan/code/chipyard/generators/chipyard/src/main/scala/IOBinders.scala' because the SemanticDB file '/Users/tynan/code/chipyard/.bloop/chipyard/bloop-bsp-clients-classes/classes-Metals-q4mXIdS8Sku5WWztCaAHqQ==/META-INF/semanticdb/generators/chipyard/src/main/scala/IOBinders.scala.semanticdb' doesn't exist. There can be many reasons for this error.

What is the expected behavior?

I should be able to find type definitions, variable definitions/references across files/modules/subprojects.

Other information

No idea if it's best to discuss this problem in the Chipyard repo, but this should be a good place to start. There are a few especially concerning notes in metals.log, but it's hard for me to know which of them are serious and which are red herrings.

2021.09.19 10:58:29 WARN  Missing project 'firrtl' may cause compilation issues in project 'firrtl_interpreter'
2021.09.19 10:58:29 WARN  Missing projects 'targetutils', 'chisel', may cause compilation issues in project 'hardfloat'
2021.09.19 10:58:29 WARN  Missing project 'chisel' may cause compilation issues in project 'iocell'

These could explain the fact that I can't see definitions of types like UInt.

Anecdotally, I seem to recall that I was able to get the rocket-chip repo working quite well with metals a month ago or so, but I didn't do a lot of experimentation.

This is a Scala compiler error, but I have no idea how much it actually affects Metals:

2021.09.19 10:58:51 ERROR 
  mergePrefixAndArgs(List((tpe: <error>)firrtl_interpreter.ConcreteUInt <and> (newWidth: Int)firrtl_interpreter.ConcreteUInt, (tpe: <error>)firrtl_interpreter.ConcreteSInt <and> (newWidth: Int)firrtl_interpreter.ConcreteSInt, (tpe: <error>)firrtl_interpreter.ConcreteClock <and> (width: Int)firrtl_interpreter.ConcreteClock), covariant, Depth(0)): unsupported tps
     while compiling: /Users/tynan/code/chipyard/tools/firrtl-interpreter/src/main/scala/firrtl_interpreter/LoFirrtlExpressionEvaluator.scala
        during phase: typer
     library version: version 2.12.10
    compiler version: version 2.12.10
  reconstructed args: -bootclasspath /Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.10/scala-library-2.12.10.jar -deprecation -unchecked -classpath /Users/tynan/code/chipyard/.bloop/firrtl_interpreter/bloop-internal-classes/classes-Metals-q4mXIdS8Sku5WWztCaAHqQ==-xE5T6ZAkRyqpNVo7j_wyhw==:/Users/tynan/code/chipyard/.bloop/firrtl_interpreter/scala-2.12/classes-empty-firrtl_interpreter:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7.1/antlr4-runtime-4.7.1.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.4.0/protobuf-java-3.4.0.jar:/Users/tynan/.sbt/boot/scala-2.12.12/lib/scala-reflect.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/github/scopt/scopt_2.12/3.7.1/scopt_2.12-3.7.1.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/net/jcazevedo/moultingyaml_2.12/0.4.2/moultingyaml_2.12-0.4.2.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/json4s/json4s-native_2.12/3.6.9/json4s-native_2.12-3.6.9.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/commons/commons-text/1.8/commons-text-1.8.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-jline/2.12.1/scala-jline-2.12.1.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/github/nscala-time/nscala-time_2.12/2.22.0/nscala-time_2.12-2.22.0.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/yaml/snakeyaml/1.26/snakeyaml-1.26.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/json4s/json4s-core_2.12/3.6.9/json4s-core_2.12-3.6.9.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/joda-time/joda-time/2.10.1/joda-time-2.10.1.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/joda/joda-convert/2.2.0/joda-convert-2.2.0.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/json4s/json4s-ast_2.12/3.6.9/json4s-ast_2.12-3.6.9.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/json4s/json4s-scalap_2.12/3.6.9/json4s-scalap_2.12-3.6.9.jar:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/2.8/paranamer-2.8.jar -Xplugin:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scalamacros/paradise_2.12.10/2.1.1/paradise_2.12.10-2.1.1.jar -Xplugin:/Users/tynan/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scalameta/semanticdb-scalac_2.12.10/4.3.22/semanticdb-scalac_2.12.10-4.3.22.jar -Xplugin-require:semanticdb -Xsource:2.11.0 -Yrangepos -Ycache-plugin-class-loader:last-modified -P:semanticdb:sourceroot:/Users/tynan/code/chipyard -P:semanticdb:failures:warning -P:semanticdb:synthetics:on

  last tree to typer: Ident(other)
       tree position: line 337 of /Users/tynan/code/chipyard/tools/firrtl-interpreter/src/main/scala/firrtl_interpreter/LoFirrtlExpressionEvaluator.scala
            tree tpe: <error>
              symbol: value other
   symbol definition: val other: <error> (a TermSymbol)
      symbol package: firrtl_interpreter
       symbol owners: value other -> value result -> method evaluate -> class LoFirrtlExpressionEvaluator
           call site: method evaluate in class LoFirrtlExpressionEvaluator in package firrtl_interpreter

== Source file context for tree position ==

   334               case SIntType(IntWidth(w)) => Concrete.randomSInt(w.toInt)
   335               case ClockType             => Concrete.randomClock()
   336               case other =>
   337                 throw InterpreterException(s"ValidIf found unsupported type: $other")
   338             }
   339           }
   340         case DoPrim(op, args, const, tpe) =>
davidbiancolin commented 3 years ago

I suspect that maybe the sriracha plugin isn't playing nice with bloop? At one point we supported a native bloop build in CY, you could try going back to the point at which we removed it and see if you could get VSCode to work from there.

I'm assuming metal's is calling "sbt bloopInstall" based on some default configuration? So if you go back in time you can see what we had to do to get bloopInstall to work, and maybe teach metals the right thing to do instead.

davidbiancolin commented 3 years ago

Sriracha is the plugin we're using to support both source and published dependencies of chisel & firrtl, hence their missing *.jsons.

tymcauley commented 3 years ago

Thanks for taking a look @davidbiancolin! Ya I remember #613, although it looks like the changes you needed to make to support bloop were mostly:

I gather that Metals is essentially supposed to take care of all of that. However, the idea of sbt-sriracha not playing nicely with Metals seems very plausible.

To explore this further, I checked out aad28dccdb798e7ee367bde9eed3520e06d4fc31 (the commit that merged #613 into dev), cleaned out the repo, and ran Metals through VSCode again. The results were definitely an improvement, and there were fewer errors. Again looking at IOBinders.scala, I was able to go-to-definition for Data on line 49, which correctly took me to the definition in tools/chisel3/core/src/main/scala/chisel3/Data.scala. However, I still get "No definition found" for types like AXI4Bundle, AXI4SlaveNode, IOCell, etc. Also, if I try to go-to-definition on a val, such as port on line 132, I'd expect it to take me to line 128, but again I got "No definition found for 'port'".

In case it's useful, here's the Metals log file from this experiment: metals.log

davidbiancolin commented 3 years ago

The no definition found thing is symptomatic of an earlier compile problem. Now it seems like the scala compiler itself is out when it's trying to compile the firrtl interpreter.

Did you try this in a fresh repo?

tymcauley commented 3 years ago

Yup, I just tried this again in a fresh repo (clone chipyard, checkout aad28dc, init submodules, populate .sbtopts, open VS Code), and got the same results.

I also tried commenting out that block of code in firrtl_interpreter which resulted in the Scala compiler crash, and the Metals log doesn't have any more concerning errors/warnings in it. At this point, when I look at the Metals Errors/Warnings (in the "Problems" tab in VS Code), the vast majority of the errors are in firrtl_interpreter and treadle, and the root cause is failing to resolve the firrtl package. Since the firrtl imports fail, there are thousands of errors failing to resolve all of the types imported from firrtl. I also saw that barstools and hardfloat had a few errors that also originate in failing to import the firrtl package. Other than that, there are just some warnings within the chisel3 package.

davidbiancolin commented 3 years ago

Right, part of why i thought it might be sriracha in the first place was because chisel.json and firrtl.json weren't being generated during bloopInstall. This too appears to be the case on that earlier version (hence your firrtl imports failing).

I'd need to have a closer look at the build.sbt, but figuring out why those json files aren't being generated is the crux of the issue.

tymcauley commented 3 years ago

Ah, this is a good clue, I'll look into this more: https://github.com/scalacenter/bloop/issues/675

tymcauley commented 3 years ago

Well, all I could find through GitHub-issue-spelunking was references to this bloop option. I tried enabling it, but the results weren't any different. Then I noticed that it looks like this option is enabled by default, so it makes sense that the outcome is the same. Anyway, that doesn't seem to resolve the issue.

tymcauley commented 3 years ago

So, I noticed this PR was merged into Rocket Chip recently: https://github.com/chipsalliance/rocket-chip/pull/2846

If that's the dependency-tracking plan going forward for Chipyard as well, then I don't know how much it makes sense to try and fix the interaction between sbt-sriracha and Metals. Any ideas if Chipyard is also going to migrate towards non-source dependencies for chisel/firrtl?

davidbiancolin commented 3 years ago

That is the plan. It is a substantially bigger under taking here though.

alphabetc1 commented 2 years ago

Same error here, it looks like that metals only load project in chipyard/build.sbt, but firrtl locates in chipyard/tools/firrtl/build.sbt.The same for project Chisel. Is there any way to source firrtl/Chisel?

tymcauley commented 2 years ago

I think it's probably not worthwhile to spend a lot of time debugging this until #1054 is merged into dev. That PR should significantly improve this situation.

tymcauley commented 2 years ago

So I actually just tried checking out the use-published-deps branch and loading it in VS Code, and things are indeed much improved. I can hover and get definitions for almost every single symbol (including Chisel/FIRRTL types), but there are a few exceptions.

Here are some of the warning messages I saw in the Metals log:

WARN  Missing project 'targetutils' may cause compilation issues in project 'hardfloat'
WARN  Missing project 'targetutils' may cause compilation issues in project 'sha3'

There was only one compilation error message from bloop, which was on line 106 of generators/sha3/src/main/scala/dpath.scala: not found: value midas.

It looks like since midas and firesimLib are still ProjectRefs in build.sbt, those might not be working properly. I don't see a .bloop/*.json file created for either of those projects. Here is the log of all the .bloop/*.json files that were created:

2021.12.08 07:48:07 INFO  [success] Generated .bloop/iocell.json
2021.12.08 07:48:07 INFO  [success] Generated .bloop/iocell-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/chipyardRoot.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/chipyardRoot-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/api-config-chipsalliance.json
2021.12.08 07:48:08 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:08 INFO  [success] Generated .bloop/api-config-chipsalliance-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/rocketMacros.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/rocketMacros-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/mdf.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/mdf-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/rocketConfig.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/rocketConfig-test.json
2021.12.08 07:48:08 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:08 INFO  [success] Generated .bloop/chisel_testers-test.json
2021.12.08 07:48:08 INFO  [success] Generated .bloop/chisel_testers.json
2021.12.08 07:48:08 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:09 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:09 INFO  [success] Generated .bloop/hardfloat-test.json
2021.12.08 07:48:09 INFO  [success] Generated .bloop/hardfloat.json
2021.12.08 07:48:09 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:09 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:11 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:11 INFO  [success] Generated .bloop/barstoolsMacros.json
2021.12.08 07:48:11 INFO  [success] Generated .bloop/barstoolsMacros-test.json
2021.12.08 07:48:11 INFO  [success] Generated .bloop/dsptools-test.json
2021.12.08 07:48:11 INFO  [success] Generated .bloop/dsptools.json
2021.12.08 07:48:11 INFO  [success] Generated .bloop/rocketchip.json
2021.12.08 07:48:11 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:11 INFO  [success] Generated .bloop/rocketchip-test.json
2021.12.08 07:48:11 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:11 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sifive_cache.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sifive_cache-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/ibex.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/ibex-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/nvdla.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/nvdla-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sodor.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/cva6-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sodor-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/cva6.json
2021.12.08 07:48:13 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sifive_blocks-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sifive_blocks.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sha3.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/sha3-test.json
2021.12.08 07:48:13 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:13 INFO  [success] Generated .bloop/rocket-dsp-utils-test.json
2021.12.08 07:48:13 INFO  [success] Generated .bloop/rocket-dsp-utils.json
2021.12.08 07:48:14 INFO  [success] Generated .bloop/testchipip.json
2021.12.08 07:48:14 INFO  [success] Generated .bloop/testchipip-test.json
2021.12.08 07:48:14 INFO  [success] Generated .bloop/hwacha.json
2021.12.08 07:48:14 INFO  [success] Generated .bloop/hwacha-test.json
2021.12.08 07:48:15 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:15 INFO  [success] Generated .bloop/icenet-test.json
2021.12.08 07:48:15 INFO  [success] Generated .bloop/icenet.json
2021.12.08 07:48:15 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:15 INFO  [success] Generated .bloop/fpga_shells.json
2021.12.08 07:48:15 INFO  [success] Generated .bloop/fpga_shells-test.json
2021.12.08 07:48:15 INFO  [success] Generated .bloop/boom.json
2021.12.08 07:48:15 INFO  [success] Generated .bloop/boom-test.json
2021.12.08 07:48:15 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:16 INFO  [success] Generated .bloop/gemmini.json
2021.12.08 07:48:16 INFO  [success] Generated .bloop/gemmini-test.json
2021.12.08 07:48:16 INFO  [success] Generated .bloop/tracegen.json
2021.12.08 07:48:16 INFO  [success] Generated .bloop/tracegen-test.json
2021.12.08 07:48:16 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:18 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:18 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:18 INFO  [success] Generated .bloop/chipyard-test.json
2021.12.08 07:48:18 INFO  [success] Generated .bloop/chipyard.json
2021.12.08 07:48:18 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:18 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:20 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:20 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:20 INFO  [success] Generated .bloop/fpga_platforms-test.json
2021.12.08 07:48:20 INFO  [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
2021.12.08 07:48:20 INFO  [success] Generated .bloop/fpga_platforms.json
2021.12.08 07:48:20 INFO  [success] Generated .bloop/firechip-test.json
2021.12.08 07:48:20 INFO  [success] Generated .bloop/firechip.json
2021.12.08 07:48:20 INFO  [success] Generated .bloop/tapeout-test.json
2021.12.08 07:48:20 INFO  [success] Generated .bloop/tapeout.json
2021.12.08 07:48:20 INFO  [success] Total time: 22 s, completed Dec 8, 2021, 7:48:20 AM
davidbiancolin commented 2 years ago

When we did have native bloop support, IIRC i had to separately run bloopInstall the jsons from both chipyard and FireSim. I think in VSCode you should be able to provide a different build server intialization script to do the slightly more complicated multi-step bloopInstall -- i'd suggest looking at that.

abejgonzalez commented 2 years ago

@tymcauley Is this better now that the Chisel 3.5 / publish-deps stuff is merged in?

tymcauley commented 2 years ago

@tymcauley Is this better now that the Chisel 3.5 / publish-deps stuff is merged in?

It looks like this are the same as described in this comment: https://github.com/ucb-bar/chipyard/issues/986#issuecomment-989079398

Which is to say, working in any project that isn't the actual chipyard generator works great. However, code navigation inside of the chipyard generator isn't awesome. It works within a single file, but not across files inside the chipyard generator.

davidbiancolin commented 2 years ago

Now that we have this stuff merged in i'll take the time in the next few weeks to figure out what's up. At SiFive we're using VSCode for most of our dev so i want to get this working in FireSim (and thus Chipyard) for my colleagues.

davidbiancolin commented 2 years ago

So i got vscode X metals to work under FireSim-as-top, and i think i had the same problem you had @tymcauley. If bloop isn't installed (listed as a plugin in plugins.sbt), the metals plugin will generally install it for you. However, it's not sufficient to do this just in FireSim or just in Chipyard, it needs to be done in both. Of course, VSCode doesn't know to do that. The workaround is to just add Bloop as a plugin to both FireSIm and Chipyard and tell VSCode that Bloop already installed in the metals plugin.

In FireSim as top, i had to use a slightly different SBT launch command but that may not be true of Chipyard.