search

chipsalliance / firrtl

Flexible Intermediate Representation for RTL
https://www.chisel-lang.org/firrtl/
Apache License 2.0
731 stars 177 forks source link

Java OutOfMemoryErrors... during Infer Widths? #222

Closed ccelio closed 8 years ago

ccelio commented 8 years ago

Something in the last 1-2 weeks seems to have broke FIRRTL with regards to java GCC memory limit in using BOOM - I believe this commit (76684cbeb8b5d46138185432bc323a5de2eb7f59).

It's the oldest commit along the mainline (76684cbeb8b5d46138185432bc323a5de2eb7f59) I can replicate the error with, and thankfully it also still prints out the compiler passes so I can see where it is. Each pass was really really really slow: 38 seconds for expand-whens for example. It usually takes 2.5 minutes to get from "make" to a compiled emulator for BOOM. But this one crashes out at 7-8 minutes during InferWidths.

Trying to go to older FIRRTLs just gives me gives lower types errors... thanks to trying to update to latest rocket-chip. But that's not why I'm here.

Here's the failing output:

[info] [24.422] Done elaborating.
[success] Total time: 28 s, completed Aug 8, 2016 1:45:50 AM
mv /scratch/celio/boom-chip/vsim/generated-src/Top.fir /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.fir
mkdir -p ./output
ln -fs /nscratch/celio/install/riscv/riscv64-unknown-elf/share/riscv-tests/isa/rv64ui-p-simple output/rv64ui-p-simple
mkdir -p /scratch/celio/boom-chip/vsim/generated-src/
echo "\`ifndef CONST_VH" > /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.vh
ar rcs libdramsim.a /scratch/celio/boom-chip/dramsim2/AddressMapping.o /scratch/celio/boom-chip/dramsim2/Bank.o /scratch/celio/boom-chip/dramsim2/BankState.o /scratch/celio/boom-chip/dramsim2/BusPacket.o /scratch/celio/boom-chip/dramsim2/CommandQueue.o /scratch/celio/boom-chip/dramsim2/IniReader.o /scratch/celio/boom-chip/dramsim2/MemoryController.o /scratch/celio/boom-chip/dramsim2/MemorySystem.o /scratch/celio/boom-chip/dramsim2/MultiChannelMemorySystem.o /scratch/celio/boom-chip/dramsim2/Rank.o /scratch/celio/boom-chip/dramsim2/SimulatorObject.o /scratch/celio/boom-chip/dramsim2/TraceBasedSim.o /scratch/celio/boom-chip/dramsim2/Transaction.o
java -Xmx4G -Xss16M -XX:MaxPermSize=2048M -cp /scratch/celio/boom-chip/firrtl/utils/bin/firrtl.jar firrtl.Driver -i /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.fir -o /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.v -X verilog
echo "\`define CONST_VH" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.vh
echo "#ifndef __CONST_H__" > /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.h
sed -r 's/\(([A-Za-z0-9_]+),([A-Za-z0-9_]+)\)/`define \1 \2/' /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.prm >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.vh
echo "#define __CONST_H__" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.h
sed -r 's/\(([A-Za-z0-9_]+),([A-Za-z0-9_]+)\)/#define \1 \2/' /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.prm >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.h
echo "\`define TBVFRAG \"Top.BOOML1Config.tb.vfrag\"" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.vh
echo "\`endif // CONST_VH" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.vh
echo "#define TBFRAG \"Top.BOOML1Config.tb.cpp\"" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.h
echo "#endif // __CONST_H__" >> /scratch/celio/boom-chip/vsim/generated-src/consts.BOOML1Config.h
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
java -Xmx4G -Xss16M -XX:MaxPermSize=2048M -cp /scratch/celio/boom-chip/firrtl/utils/bin/firrtl.jar firrtl.Driver -i /scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.fir -o /scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.v -X verilog
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOf(Arrays.java:2367)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
    at java.lang.StringBuilder.append(StringBuilder.java:132)
    at java.lang.StringBuilder.append(StringBuilder.java:128)
    at scala.StringContext.standardInterpolator(StringContext.scala:125)
    at scala.StringContext.s(StringContext.scala:95)
    at firrtl.Serialize.serialize(Serialize.scala:81)
    at firrtl.Serialize$.serialize(Serialize.scala:39)
    at firrtl.ir.FirrtlNode.serialize(IR.scala:41)
    at firrtl.WrappedExpression.hashCode(WIR.scala:104)
    at scala.runtime.ScalaRunTime$.hash(ScalaRunTime.scala:206)
    at scala.collection.mutable.HashTable$HashUtils$class.elemHashCode(HashTable.scala:409)
    at scala.collection.mutable.LinkedHashMap.elemHashCode(LinkedHashMap.scala:49)
    at scala.collection.mutable.HashTable$class.findEntry(HashTable.scala:132)
    at scala.collection.mutable.LinkedHashMap.findEntry(LinkedHashMap.scala:49)
    at scala.collection.mutable.LinkedHashMap.get(LinkedHashMap.scala:65)
    at scala.collection.MapLike$class.contains(MapLike.scala:150)
    at scala.collection.AbstractMap.contains(Map.scala:59)
    at firrtl.passes.ExpandWhens$$anonfun$firrtl$passes$ExpandWhens$$expandWhens$2$3.apply(ExpandWhens.scala:139)
    at firrtl.passes.ExpandWhens$$anonfun$firrtl$passes$ExpandWhens$$expandWhens$2$3.apply(ExpandWhens.scala:137)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
    at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
    at firrtl.passes.ExpandWhens$.firrtl$passes$ExpandWhens$$expandWhens$2(ExpandWhens.scala:137)
    at firrtl.passes.ExpandWhens$$anonfun$firrtl$passes$ExpandWhens$$expandWhens$2$4.apply(ExpandWhens.scala:176)
    at firrtl.passes.ExpandWhens$$anonfun$firrtl$passes$ExpandWhens$$expandWhens$2$4.apply(ExpandWhens.scala:176)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
make: *** [/scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.v] Error 1

Another failure gave me a Java heap space error:

java -Xmx16G -Xss32M -XX:MaxPermSize=81928M -cp /scratch/celio/boom-chip/firrtl/utils/bin/firrtl.jar firrtl.Driver -i /scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.fir -o /scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.v -X verilog
[INFO] Starting ANTLR Parser
[INFO] Finished ANTLR Parser
[INFO] ANTLR Parser took 0.0 ms

...

[INFO] Starting Resolve Genders
[INFO] Finished Resolve Genders
[INFO] Resolve Genders took 17339.0 ms

[INFO] Starting Infer Widths
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at scala.collection.mutable.HashTable$class.resize(HashTable.scala:251)
    at scala.collection.mutable.HashTable$class.scala$collection$mutable$HashTable$$addEntry0(HashTable.scala:154)
    at scala.collection.mutable.HashTable$class.findOrAddEntry(HashTable.scala:166)
    at scala.collection.mutable.LinkedHashMap.findOrAddEntry(LinkedHashMap.scala:49)
    at scala.collection.mutable.LinkedHashMap.put(LinkedHashMap.scala:71)
    at scala.collection.mutable.LinkedHashMap.$plus$eq(LinkedHashMap.scala:89)
    at scala.collection.mutable.LinkedHashMap.$plus$eq(LinkedHashMap.scala:49)
    at scala.collection.mutable.MapLike$class.update(MapLike.scala:88)
    at scala.collection.mutable.AbstractMap.update(Map.scala:80)
    at firrtl.passes.InferTypes$.firrtl$passes$InferTypes$$infer_types_s$1(Passes.scala:200)
    at firrtl.passes.InferTypes$$anonfun$firrtl$passes$InferTypes$$infer_types_s$1$3.apply(Passes.scala:213)
    at firrtl.passes.InferTypes$$anonfun$firrtl$passes$InferTypes$$infer_types_s$1$3.apply(Passes.scala:213)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.immutable.List.map(List.scala:285)
    at firrtl.Mappers$StmtMagnet$$anon$4.map(Mappers.scala:44)
    at firrtl.Mappers$StmtMap.map(Mappers.scala:95)
    at firrtl.passes.InferTypes$.firrtl$passes$InferTypes$$infer_types_s$1(Passes.scala:213)
    at firrtl.passes.InferTypes$$anonfun$firrtl$passes$InferTypes$$infer_types_s$1$3.apply(Passes.scala:213)
    at firrtl.passes.InferTypes$$anonfun$firrtl$passes$InferTypes$$infer_types_s$1$3.apply(Passes.scala:213)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at firrtl.Mappers$StmtMagnet$$anon$4.map(Mappers.scala:44)
    at firrtl.Mappers$StmtMap.map(Mappers.scala:95)
    at firrtl.passes.InferTypes$.firrtl$passes$InferTypes$$infer_types_s$1(Passes.scala:213)
    at firrtl.passes.InferTypes$$anonfun$firrtl$passes$InferTypes$$infer_types_s$1$3.apply(Passes.scala:213)
make: *** [/scratch/celio/boom-chip/emulator/generated-src/Top.BOOMConfig.v] Error 1

Asides:

I'd appreciate some sort of statistics output from FIRRTL telling me how many "nodes" a design used. It'd help keep an eye on exploding node counts from either broken Chisel/FIRRTL passes or poor Coding style. And it may help highlight when something like this starts to crop up.

Also, how can I turn back on the FIRRTL pass print outs? When this stuff is failing I'd like to know where I'm stuck at.

jackkoenig commented 8 years ago

Sorry that the pass print outs isn't more user friendly. You can change the level by modifying src/main/resources/logback.xml. I spent some time trying to figure out how to globally change the level programmatically (ie. with a command-line option), but couldn't figure it out. I'll try to take a look again soon.

As for a pass to print the number of nodes, I totally agree. This would help us detect issues like #200 and try to fix them before people with large designs get stuck.

ccelio commented 8 years ago

This OutOfMemory error shows up when trying to compile BOOM after enabling BOOM's many "printf" statements to be emitted.

ccelio commented 8 years ago

I can get it to compile to Verilog if I increase the Java limits to absurd levels:

java -Xmx16G -Xss32M -XX:MaxPermSize=4192M 
  -cp /scratch/celio/boom-chip/firrtl/utils/bin/firrtl.jar firrtl.Driver 
  -i /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.fir 
  -o /scratch/celio/boom-chip/vsim/generated-src/Top.BOOML1Config.v -X verilog`

But that wasn't required before, and I'm noticing significant increases in compile times now. For example, BOOM without printfs takes 2x the amount of time to generate a Verilog file than it used to.

azidar commented 8 years ago

Can I look at your .fir file?

ccelio commented 8 years ago

To reproduce:

git clone https://github.com/ucb-bar/rocket-chip.git boom-chip
cd boom-chip
git checkout boom
git submodule update --init
cd emulator
time make CONFIG=BOOMConfig
ls generated-src
ccelio commented 8 years ago

To reproduce the "printf" issue that makes the graph bigger...

vim boom-chip/boom/src/main/scala/const.scala
modify line 24, set "val DEBUG_PRINTF=true".
azidar commented 8 years ago

Didn't build, getting the following error:

[error] /Users/cusgadmin/code/scala/boom-chip/rocket/src/main/scala/nbdcache.scala:10: object util is not a member of package uncore
[error] import uncore.util._
ccelio commented 8 years ago

By starting from a clean repository checkout and following the above instructions, I am unable to reproduce your error message.

The rocket-chip commit is bb1c4e2ef9014a0f55f21937c6e02a35f80d43cd. The boom submodule commit is 75bba38ba406f4e6fb3b1f055cf72bd1619587ca.

donggyukim commented 8 years ago

256