Cal-Poly-RAMP / tapeout-ci-2311

caravel-user repository for November 6, 2023 tapeout
Apache License 2.0
1 stars 1 forks source link

Magic DRC During MPW Precheck Hangs Indefinitely #10

Closed Peter-Herrmann closed 8 months ago

Peter-Herrmann commented 8 months ago

When reaching the MPW precheck, the check is successful up to the Magic DRC check. While this step does not fail, it seems to enter a faulted state.The process will consume as high as 311GB of RAM (filling physical RAM and the entire swapfile), and run a single core at 100% (or as high as it can limited by page faults). It will run in this state seemingly forever, longer than 8 hours.

Per this document and this thread, it is possible that there is an issue with the SRAM macros during the DRC check. In both these cases, it is recommended to use the latest version of the macro directly from the PDK provided by efabless.


If I use the PDK version of the SRAM, using the following in config.json:

"VERILOG_FILES_BLACKBOX": "pdk_dir::libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v",
"EXTRA_LEFS": "pdk_dir::libs.ref/sky130_sram_macros/lef/sky130_sram_2kbyte_1rw1r_32x512_8.lef",
"EXTRA_LIBS": "pdk_dir::libs.ref/sky130_sram_macros/lib/sky130_sram_2kbyte_1rw1r_32x512_8_TT_1p8V_25C.lib",
"EXTRA_GDS_FILES": "pdk_dir::libs.ref/sky130_sram_macros/gds/sky130_sram_2kbyte_1rw1r_32x512_8.gds",

Then I recieve linter errors (which can be ignored), and STA errors (which cannot be ignored):

Error while reading /home/prherrma/runners/r0/_work/pdk/sky130A/libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v:
Make sure that this a gate-level netlist not an RTL file
You can add the following comment '/// sta-blackbox' in the file to skip it and blackbox the modules inside if needed.

Examples:


If I provide a modified version of the verilog blackbox file, with /// sta-blackbox as recommended, then I can avoid all linter and STA errors, but when reaching the Magic DRC check it will hang in a seemingly infinite loop and consume all of the system resources. image

Examples:

Peter-Herrmann commented 8 months ago

Another interesting symptom: These commits got different results during detailed routing, despite theoretically using the exact same files. Instead of using a relative path (dir::../../../../pdk/), I use the environment variable pdk_dir::, and I omitted the VERILOG_INCLUDE_DIRS config.

By doing this, I somehow ended up with different detailed routing results, which seems to indicate an actual difference in what is being put on the die, or at least the actions taken during optimization.

EDIT Similarly, 8705297 changed nothing but re-adding the VERILOG_INCLUDE_DIRS and it ended up with yet another violation count.

Peter-Herrmann commented 8 months ago

I'm basically stuck between two options:

  1. I implement using the provided PDK versions, which causes STA checks to fail
  2. I implement using a modified blackbox, which causes the Magic DRC to hang indefinitely.

To move forward, I need to do one of the following:

Peter-Herrmann commented 8 months ago

This was found to be a bug with the MPW precheck itself, described in issue #247. Currently awaiting a fix to be merged into the MPW precheck in PR #248

Peter-Herrmann commented 8 months ago

Found that the Makefile target make pdk-with-volare was using an old, hard-coded version of the PDK:

ifeq ($(PDK),sky130A)
    SKYWATER_COMMIT=f70d8ca46961ff92719d8870a18a076370b85f6c
    export OPEN_PDKS_COMMIT?=78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc
    export OPENLANE_TAG?=2023.07.19
    MPW_TAG ?= mpw-9e

This vesion of the PDK (78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc) was missing .maglef files for the SRAM we are using, and the DRC script is not set up to handle that in the MPW precheck. After setting the PDK version to 559a117b163cef2f920f33f30f6f690aa0b47e4c (latest), the DRC check completed.

Peter-Herrmann commented 8 months ago

After updating the OPENLANE_TAG, I recieved the following warning:

Warning: : OpenLane may not function properly: open_pdks 559a117b163cef2f920f33f30f6f690aa0b47e4c
The version of open_pdks used in building the PDK does not match the version OpenLane was tested on (installed: 559a117b163cef2f920f33f30f6f690aa0b47e4c, tested: dd7771c384ed36b91a25e9f8b314355fc26561be)

Simply updating the version in the Makefile was not sufficient. In order to get the tested version, I needed to explicitly enable the tested version with volare in the ci.yml:

      - name: Build PDK
        run: |
          make pdk-with-volare
          python3 -m pip install --upgrade --no-cache-dir volare
          volare enable --pdk sky130 dd7771c384ed36b91a25e9f8b314355fc26561be
          volare ls --pdk sky130