commit #0b7fef44f removes SBT, requires mill

[gsomlo]

Type of issue: bug report

Impact: unknown

If the current behavior is a bug, please provide the steps to reproduce the problem:

make ... -C rocket-chip/vsim verilog CONFIG=<model> fails (is no longer self-contained)

What is the current behavior?

...
/bin/bash: line 1: mill: command not found
...

What is the expected behavior?

make command successfully builds inside self-contained rocket-chip + submodules repo tree.

Please tell us about your environment:

- version: 50adbdb3e
- OS: `Linux 6.4.12-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 23 17:46:49 UTC 2023 x86_64 GNU/Linux`

What is the use case for changing the behavior?

I'm not sure if disabling SBT in favor of mill was deliberate or accidental (I'm aware the commit was lurking around in the dev branch for a while now :) ). However, if it's deliberate, it'd be nice if mill were included as a submodule or somehow otherwise made a "transparent" part of the build process within the rocket-chip repo...

[gsomlo]

Also, if I install and configure mill like so:

git clone https://github.com/com-lihaoyi/mill
mill/mill --help  # downloads to ${HOME}/.cache/mill/download/0.11.0
export MILL=${HOME}/.cache/mill/download/0.11.0

then

make ... -C rocket-chip/vsim verilog CONFIG=...

I get an error:

make: Entering directory '~/rocket-chip/vsim'
cd ~/rocket-chip && ~/.cache/mill/download/0.11.0 rocketchip.assembly
[build.sc] [44/49] cliImports 
Cannot resolve rocketchip.assembly. Try `mill resolve rocketchip._` to see what's available.
make: *** [~/rocket-chip/Makefrag:47: ~/rocket-chip/out/rocketchip/assembly.dest/out.jar] Error 1
make: Leaving directory '~/rocket-chip/vsim'

This did not use to happen a few months ago when I (accidentally :) ) tested the dev branch and figured out how to use mill...

[n-kremeris]

I am facing the same issue as @gsomlo.

I have also tried using different versions of mill (e.g. "DEFAULT_MILLVERSION=0.11.2 make" in rocket-chip/emulator), while still observing the same error no matter which version I used. "Cannot resolve rocketchip.assembly. Try `mill resolve rocketchip.` to see what's available. "

commit 50adbdb3e4e18c2b3de57693323f4174b60f9767

[HakamAtassi]

Me as well. I have reverted to commit [eee99e8] and things seem to be going more smoothly, so far.

Nevermind, I seem to be having issues across the board. I'm unable to build rocketchip.

[jerryz123]

Unfortunately, the standalone rocket-chip build support has been deprecated and removed, due to lack of developer resources to maintain that feature. To build and test SoCs with rocket-chip, users should seek out SoC frameworks.

Two options include :

• https://github.com/ucb-bar/chipyard, which provides a comprehensive mono-repo for hardware design
• https://github.com/chipsalliance/playground, which uses mill+nix for a lighter-weight, and more modular development environment

[gsomlo]

On Mon, Sep 18, 2023 at 10:34:54PM -0700, Jerry Zhao wrote:

Unfortunately, the standalone rocket-chip build support has been deprecated and removed, due to lack of developer resources

While I find that extremely disappointing, I fully understand about limited resources available for maintenance.

My own ToDo list has been known to grow faster than I can address tasks currently on it. In fact, based on this decision, one such high-priority item: "learn enough Chisel to understand (and maybe eventually contribute to) Rocket" can now be safely scratched off that list, freeing up valuable time for other things I'm supposed to be working on... :D

[n-kremeris]

Unfortunately, the standalone rocket-chip build support has been deprecated and removed, due to lack of developer resources to maintain that feature. To build and test SoCs with rocket-chip, users should seek out SoC frameworks.

Two options include :

• https://github.com/ucb-bar/chipyard, which provides a comprehensive mono-repo for hardware design
• https://github.com/chipsalliance/playground, which uses mill+nix for a lighter-weight, and more modular development environment

Hi Jerry and thanks for the info. Could you elaborate a little bit about what exactly is removed? If i understand correctly, "removal of standalone build support" is actually more like the build configs from sbt haven't been moved to mill?

I'm currently ramping up on scala/chisel with some mill on the side, so unless there are some standalone-build breaking changes in the chisel code itself, I might try and port the standalone build from sbt to mill.

[jerryz123]

If i understand correctly, "removal of standalone build support" is actually more like the build configs from sbt haven't been moved to mill?

The SBT build files, and the Makefile flow which relied on that, no longer works.

It is still entirely possible to emit verilog from this repo using mill, unfortunately the README hasn't caught up with these changes. @sequencer can you or someone on your end update the README?

I might try and port the standalone build from sbt to mill.

To be clear, the part that is broken is the standalone makefile-to-verilog flow. This would be worth bringing up to date.

[jerryz123]

@gsomlo @n-kremeris , I see that you guys are looking at updating rocket-chip support within Litex.

We are focusing efforts towards making rocket-chip usable as a library, rather than a standalone repository. Maintaining whatever is necessary to make rocket-chip-Litex integration easy is work I, along with the other active maintainers, would be willing to do.

[gsomlo]

@jerryz123 Thanks much for the clarification! Short of using SBT, I would be perfectly OK going with a mill-based process similar to what I outlined in https://github.com/chipsalliance/rocket-chip/issues/3483#issuecomment-1716426109 (above).

However, the steps shown there don't seem to be working at the moment either (I'm getting Cannot resolve rocketchip.assembly. Try ``mill resolve rocketchip._`` to see what's available.). If whatever causes that error could be fixed (whether by providing an updated set of steps for using mill to make verilog, or by fixing whatever's currently broken within the rocket-chip repo to to get make verilog working again[1] when using mill), that would be a perfectly fair way to move forward :)

[1]: the steps shown in https://github.com/chipsalliance/rocket-chip/issues/3483#issuecomment-1716426109 did work on the dev branch of rocket-chip somewhere around the July-August time frame, but aren't working (using mill) right now on the master branch.

Thanks again, --G

[jerryz123]

Try mill rocketchip[3.6.0].assembly

I will try to find some time to restore the make verilog flow. I think make sim will be no longer be supported going forwards.

[gsomlo]

@jerryz123 -- looks like #3489 is the solution. Any idea when it might land in master (currently it has only been applied to the dev branch).

[jerryz123]

I opened the backport PR. It should get merged in as soon as the tests run.

Some developers want the dev branch to aggressively pursue newer Chisel versions, which could break backwards-compatibility for many projects which cannot bump Chisel versions so aggressively. Thus, we've adopted a backport flow, where PRs are made against dev, then backported to master.

[sequencer]

Sorry for the late reply. I'll update the readme later this week. I'm currently out of the office. For the vcs simulation flow, I'd like to provide some base BFM infrastructure in the future and use verilog+dpi TB to support both verilator and vcs.

[jerryz123]

3494 updates the Makefile + README.

More importantly, the backport PRs seem to get stuck in CI somewhere... which is blocking commits to master

[jerryz123]

We've updated the README + makefile in master so that a simple make verilog flow now works.

[jerryz123]

I jumped the gun, I'm getting confused with all these PRs. #3494 will go in imminently, and once that goes in, the backport will fix up master. This should happen within a few hours of this post.

[gsomlo]

so, I tried (against commit e3773366a, current tip of master branch):

export MILL=${HOME}/.cache/mill/download/0.11.0
make RISCV=${HOME}/RISCV -C rocket-chip verilog CONFIG=freechips.rocketchip.system.TinyConfig

and got (after a fair amount of work happened):

freechips.rocketchip.system.TestHarness,freechips.rocketchip.system.TinyConfig].mfccompiler.compile 
1 targets failed
emulator[freechips.rocketchip.system.TestHarness,freechips.rocketchip.system.TinyConfig].mfccompiler.compile java.io.IOException: Cannot run program "firtool" (in directory "/home/somlo/LITEX/pythondata-cpu-rocket/pythondata_cpu_rocket/verilog/rocket-chip/out/emulator/freechips.rocketchip.system.TestHarness/freechips.rocketchip.system.TinyConfig/mfccompiler/compile.dest"): error=2, No such file or directory
    java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
    java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
    os.proc.proc$lzycompute$1(ProcessOps.scala:134)
    os.proc.proc$1(ProcessOps.scala:128)
    os.proc.spawn(ProcessOps.scala:141)
    os.proc.call(ProcessOps.scala:80)
    millbuild.build$Emulator$mfccompiler$.$anonfun$compile$2(build.sc:152)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:71)
java.io.IOException: error=2, No such file or directory
    java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
    java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
    java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
    java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
    java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
    os.proc.proc$lzycompute$1(ProcessOps.scala:134)
    os.proc.proc$1(ProcessOps.scala:128)
    os.proc.spawn(ProcessOps.scala:141)
    os.proc.call(ProcessOps.scala:80)
    millbuild.build$Emulator$mfccompiler$.$anonfun$compile$2(build.sc:152)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:71)
make: *** [Makefile:11: verilog

I then tried one of my customized LiteX configurations:

cat >> rocket-chip/src/main/scala/subsystem/Configs.scala <<- "EOT"

    class WithMemoryDataBits(dataBits: Int) extends Config((site, here, up) => {
      case MemoryBusKey => up(MemoryBusKey, site).copy(beatBytes = dataBits/8)
    })

    class WithLitexMemPort extends Config((site, here, up) => {
      case ExtMem => Some(MemoryPortParams(MasterPortParams(
                          base = x"8000_0000",
                          size = x"8000_0000",
                          beatBytes = site(MemoryBusKey).beatBytes,
                          idBits = 4), 1))
    })

    class WithLitexMMIOPort extends Config((site, here, up) => {
      case ExtBus => Some(MasterPortParams(
                          base = x"1000_0000",
                          size = x"7000_0000",
                          beatBytes = site(SystemBusKey).beatBytes,
                          idBits = 4))
    })

    class WithLitexSlavePort extends Config((site, here, up) => {
      case ExtIn  => Some(SlavePortParams(
                          beatBytes = site(SystemBusKey).beatBytes,
                          idBits = 8,
                          sourceBits = 4))
    })
    EOT

# Configure port settings, ensure sufficient external IRQs:
cat >> rocket-chip/src/main/scala/system/Configs.scala <<- "EOT"

    class BaseLitexConfig extends Config(
      new WithLitexMemPort() ++
      new WithLitexMMIOPort() ++
      new WithLitexSlavePort ++
      new WithNExtTopInterrupts(8) ++
      new WithCoherentBusTopology ++
      new BaseConfig
    )

    class WithLitexHextConfig extends Config(
      new WithHypervisor ++
      new WithBitManip ++ new WithBitManipCrypto ++
      new WithCryptoNIST ++ new WithCryptoSM
    )

        class LitexConfig_linux_8_2 extends Config(
          new WithNBigCores(8) ++
          new WithMemoryDataBits(128) ++
          new BaseLitexConfig
        )
    EOT

make RISCV=${HOME}/RISCV -C rocket-chip verilog CONFIG=freechips.rocketchip.system.LitexConfig_linux_8_2

and got (in fairly short order):

[build.sc] [44/49] cliImports 
Cannot resolve emulator[freechips.rocketchip.system.TestHarness,freechips.rocketchip.system.LitexConfig_linux_8_2].mfccompiler.compile. Try `mill resolve emulator._` to see what's available.
make: *** [Makefile:11: verilog] Error 1

I'm guessing there's still something missing before one could successfully do a make verilog build...

Thanks again for looking into it!

[n-kremeris]

@gsomlo You can download firtool from here: https://github.com/llvm/circt/releases I managed to complete "make verilog" successfully with the default config now

[n-kremeris]

To generate the litex custom config, at least 3 changes must be made from the Rocket side of things:

1) remove underscores, e.g. LitexConfig_linux_1_1 -> LitexConfigLinux11 2) add LitexConfigLinux11 into build.sc, eg. ~lines 265-283: ("freechips.rocketchip.system.TestHarness", "freechips.rocketchip.system.LitexConfigLinux11"), 3) Change DefaultConfig to LitexConfigLinux11 in Makefile

This seems to correctly build the rocketchip with the desired Litex config. I haven't yet looked at what changes must be done to update.sh/pythondata-cpu-rocket

[gsomlo]

On Tue, Oct 03, 2023 at 03:50:21AM -0700, Norbert Kremeris wrote:

@gsomlo You can download firtool from here

Oh, great! First mill, OK. Now some other rando download -- this is quickly headed for NPM-like levels of dependency hell... :(

[seldridge]

Oh, great! First mill, OK. Now some other rando download -- this is quickly headed for NPM-like levels of dependency hell... :(

😂 That's not just any rando dependency. 😂 That's a much more (~4x conservatively) performant FIRRTL compiler that produces better, more readable Verilog output. 😉

Note that on the Chisel side, we are intending to make this download automatically handled like a normal Java dependency. @jackkoenig has an experimental approach that will download this for you from Maven. It's slightly more complicated than normal as it's a binary dependency and not a Java dependency and this has taken some time to sort out.

[sequencer]

In order to resolve the dependency issues(and avoid NPM dependency hell as well), In rocket-chip, I choose these solutions:

• for JVM stack-software, use mill for build and dependency management(believe me SBT sucks);
• for dependencies that don't provide a release, use mill to build from source: hardfloat, cde, riscv-opcodes
• for native dependencies, we use nix to manage them, and make it be able to run in different architecture and distros, (e.g. most user use x86, while I'm using an aarch64-linux, some use Ubuntu, while I'm using Arch). for native dependencies, there are two different parts:
  1. software for elaboration and emulation, e.g. circt, espresso, verilator.
  2. software for test case generation, e.g. riscv-gcc riscv-clang, riscv arch test, riscv tests.

I understand installing nix, and run a daemon for it is not quite straightforward for hardware design users, I aware that, and I'll also try to find some alternative solutions to new users(e.g. provide a docker container and release it)

I hope users understand, we reduces burdens(SBT, legacy Chisel version support, a stable API and a version based backport-flow) since

1. dependencies are hard to manage, there are different ways to manage, see other project from Google, they even depend on a shared cache. Nix may be hard to understand by new users, but it really save our time.
2. build system are hard to manage, mill does have some shortages(e.g. absolute path), but it's almost a state of the art comparing to other different build systems.(SiFive even develop their own build system https://github.com/sifive/wake)
3. We have a limited human resource to manage such a big project. For now only 2 active PhD(I and Jerry) are maintaining it(originally the entire SiFive R&D)

After all, we are still making progress, we may have some breaking change, or even introduce issues that makes some user not happy(like this) in the future, but please understand, our goal for now is improving the development efficiency to make the rocket-chip continue and alive at least.

So for most dependencies/build questions, my only suggestion is: nix develop -c mill resolve __ and read the build script to see what you can do.

[jerryz123]

2. add LitexConfigLinux11 into build.sc, eg. ~lines 265-283: ("freechips.rocketchip.system.TestHarness", "freechips.rocketchip.system.LitexConfigLinux11"),

@sequencer Is it possible to have mill parse a string for the Config as it was done before? Having to mirror the configs into the build.sc is not ideal

[sequencer]

@sequencer Is it possible to have mill parse a string for the Config as it was done before? Having to mirror the configs into the build.sc is not ideal

It's not simple, since, mill find configuration statically, there are two solutions:

1. provide these configuration from another file, e.g. "config.json", and mill read this file to generate the task graph.(See what I did in chipsalliance/t1 ;p)
2. parse Scala Config, and automatically generate the task via meta-build feature, which is not a trivial work, and I think we don't have bandwidth for it.

[gsomlo]

On Tue, Oct 03, 2023 at 02:57:05PM -0700, Jiuyang Liu wrote:

In order to resolve the dependency issues(and avoid NPM dependency hell as well), In rocket-chip, I choose these solutions:

@sequencer, @jerryz123: First off, thank you for your hard work maintaining this project. Last thing I want is to add "negative energy" from the peanut gallery, and sorry if anything I said reads that way... :)

As a "downstream consumer" of RocketChip, I (personally) value the ability to trust the output of a build process by keeping a handle on all (most?) of the "moving parts". A large part of that is ability to understand, track, and control the "main" project's dependencies.

It makes sense for a project such as Rocket to depend on a few other projects that are also "fast-evolving" in their own right (e.g., Chisel, mill, etc.).

But in order to maintain any sort of confidence in the output of a build of the project, as we descend down the tree of dependencies, things should hopefully settle down into "slow-and-stable", i.e. things that might be reasonably packaged for a distro (debian/ubuntu/fedora/etc.) and updated once every few months, if not even less frequently.

If the "fast-evolving" nature of the dependency chain is preserved too many layers deep, then we have an "everything everywhere all at once" situation that only one or two developers can reasonably keep a handle on, and anyone else might get lucky to receive a dockerfile of a snapshot of one of the developers' environments, so as to be able to ritually "cargo-cult" a rebuild, but without any chance of any control or understanding of what's going on.

I understand the latter scenario is sometimes inevitable, and that's OK, I'm happy to wait for it to stabilize once again. I wish I were enough of a Chisel/Scala ninja to offer some assistance, but that's not (yet) the case... :(

Best of luck with your degrees, that should be priority #1! :D

Cheers, --Gabriel

[n-kremeris]

@jerryz123 @sequencer

We've updated the README + makefile in master so that a simple make verilog flow now works.

Do we have a way to restore the verilator emulator binary build flow too?

[sequencer]

Do we have a way to restore the verilator emulator binary build flow too?

I don't think so, you can maintain it on your own, but not in the rocketchip upstream repo.

You can see github action, emulator works fine, It works even on an aarch64 machine, I don't think there is any reason to restore it.

[n-kremeris]

You can see github action, emulator works fine, It works even on an aarch64 machine, I don't think there is any reason to restore it.

Hi and thanks for the reply! Please could you explain a little bit more? I looked at the github actions tab, but I'm not sure where to find references to the emulator

[sequencer]

try nix develop -c mill runnable-riscv-test[freechips.rocketchip.system.TestHarness,freechips.rocketchip.system.DefaultConfig,_,_].run to run all test for DefaultConfig