Old OpenLane version can pass the detail routing but latest version cannot

AyaseErii commented 1 year ago

Description

Hello, with the same design, I happened to find that the old openlane version can pass the detail routing which will make the violations to 0, while the latest openlane cannot pass and remain several violations. I checked the openroad-app commit as well. The old openlane's openroad-app commit is 0b8b7ae255f8fbbbefa57d443949b84e73eed757 while the latest openlane's openroad commit is e036ecfaca4bc0efe88a54085efcf0f562c48a6b. Therefore, I suspect that the issue should be caused by the openroad-app, and believe the mpw-7a commit can work while the other commit cannot. If it is true, how can I find the mpw-7a based openlane and/or switch to mpw-7a with latest openlane? Simply changing the commit in openroad-app with the mpw-7a commit in the latest openlane cannot work and it will warn me to change the environment. I do not have any idea about how to change the environment. Could anyone can point me in the right direction? Thanks!

Environment

The old environment

python3 ./env.py issue-survey
Kernel: Linux v4.15.0-191-generic
Distribution: ubuntu 18.04
Python: v3.6.9 (OK)
Container Engine: docker v20.10.7 (OK)
OpenLane Git Version: a9883335a102c9ba1d862a209a72fc8b4c20c785
pip: INSTALLED
pip:venv: INSTALLED
---
PDK Version Verification Status: OK
---
Git Log (Last 3 Commits)

a988333 2022-06-18T11:53:39+02:00 Enhancements + Bugfixes to Multi-PDK Support (#1138) - Mohamed Gaber -  (HEAD -> master, tag: 2022.06.19_01.37.45)
732c856 2022-06-18T11:22:57+02:00 Fix `skip_basic_prep`, step mechanics (#1143) - Mohamed Gaber -  ()
074a92b 2022-06-14T15:01:11+02:00 Fix Magic Antenna Check (#1142) - Mohamed Gaber -  (tag: 2022.06.15_01.35.48)

The latest environment

python3 ./env.py issue-survey
Kernel: Linux v4.15.0-191-generic
Distribution: ubuntu 18.04
Python: v3.7.13 (OK)
Container Engine: docker v20.10.17 (OK)
OpenLane Git Version: 4476a58407d670d251aa0be6a55e5391bb181c4e
pip: INSTALLED
python-venv: INSTALLED
---
PDK Version Verification Status: OK
---
Git Log (Last 3 Commits)

4476a58 2022-07-29T16:03:10+02:00 Sky130 Enhancements (#1240) - Mohamed Gaber -  (HEAD -> master, tag: 2022.07.30, origin/master, origin/HEAD)
4f63c8c 2022-07-27T17:26:27+02:00 Various CI Fixes (#1235) - Mohamed Gaber -  (tag: 2022.07.28)
0b55101 2022-07-26T20:37:20+02:00 [BOT] Update openroad_app (#1219) - Openlane Bot -  (tag: 2022.07.27)
---
Git Remotes

origin  https://github.com/The-OpenROAD-Project/OpenLane.git (fetch)
origin  https://github.com/The-OpenROAD-Project/OpenLane.git (push)

Reproduction Material

No issue_repoduction directory was produced. Will upload all other files if needed.

Expected behavior

The design should pass detailed routing without violation

Logs

With latest version

[ERROR]: There are violations in the design after detailed routing.
[ERROR]: Total Number of violations is 18
[ERROR]: Flow failed.

maliberty commented 1 year ago

@donn is there a doc on how to use the latest OR with a given OL commit?

donn commented 1 year ago

@maliberty Update the OR commit in dependencies/tool_metadata.yml and invoke BUILD_IF_CANT_PULL=1 make openlane in the OpenLane root folder.

@AyaseErii If it's not too much of a hassle, can you run export CREATE_REPRODUCIBLE_FROM_SCRIPT=droute.tcl before running the flow? This should force a reproducible to be generated.

AyaseErii commented 1 year ago

@maliberty Update the OR commit in dependencies/tool_metadata.yml and invoke BUILD_IF_CANT_PULL=1 make openlane in the OpenLane root folder.

@AyaseErii If it's not too much of a hassle, can you run export CREATE_REPRODUCIBLE_FROM_SCRIPT=droute.tcl before running the flow? This should force a reproducible to be generated.

Thanks! I reproduced the issue by using the command you provided. Plz find the attached repo file: issue_reproducible.zip

When I tried to invoke BUILD_IF_CANT_PULL=1 make openlane, it showed an error that there is no module named click, then I use pip3 install click to install but it showed an message:

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: click in ./.local/lib/python3.7/site-packages (8.2.0.dev0)
Requirement already satisfied: importlib-metadata in ./.local/lib/python3.7/site-packages (from click) (4.12.0)
Requirement already satisfied: typing-extensions>=3.6.4 in ./.local/lib/python3.7/site-packages (from importlib-metadata->click) (4.3.0)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.7/site-packages (from importlib-metadata->click) (3.8.1)

I really don't know what is the issue. Any suggestions?

Thanks!

vijayank88 commented 1 year ago

@AyaseErii Share your mpw/gihub repo link

AyaseErii commented 1 year ago

@AyaseErii Share your mpw repo link

I am not sure what the mpw repo link is but I put the github repo link here.

maliberty commented 1 year ago

I just tried your reproducer with the current head of OR (b983eb1e521644fd33877c14ccd5cdde2b99aa89) and it routed just fine.

donn commented 1 year ago

@vijayank88 Let's not confuse users. It is not their responsibility to update OpenROAD in OpenLane, that exists as a developer mechanism.

Also, that particular step won't work anyway.

@maliberty It routes just fine, it just ends up with DRC errors. Does routing "just fine" include no DRC errors?

vijayank88 commented 1 year ago

@donn I just checked user repo for source config.tcl. User trying to use mpw-7a tag, but die size more than mpw pre-defined value. If they're not submitting to mpw shuttle, thought of suggesting to use latest OpenLane tag from master repo for quick fix.

vijayank88 commented 1 year ago

@donn routed just fine means routing completed with no DRC erros. With latest OR, routing completed without any violations for attached issue_reproducible

[INFO DRT-0199]   Number of violations = 0.
[INFO DRT-0267] cpu time = 00:00:01, elapsed time = 00:00:00, memory = 5806.96 (MB), peak = 6342.12 (MB)

donn commented 1 year ago

Then yeah, I need to update OR again

AyaseErii commented 1 year ago

Thanks for the help. I tried to re-build the OpenLane. However, I meet this error when I use make in the OpenLane directory after git-clone the tool.

make[1]: Entering directory '/home/htf6ry/OpenLane'
57f3f21d6e1d0403a88d7a7973f7447597aae160: Pulling from efabless/openlane
Digest: sha256:0b9a27fe51d8d709e77f653c39dd1244738885d0d39fdefd81a272824a33b46f
Status: Image is up to date for efabless/openlane:57f3f21d6e1d0403a88d7a7973f7447597aae160
docker.io/efabless/openlane:57f3f21d6e1d0403a88d7a7973f7447597aae160
make[1]: Leaving directory '/home/htf6ry/OpenLane'
rm -rf ./venv
python3 -m venv ./venv
Error: Command '['/home/htf6ry/OpenLane/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
Makefile:130: recipe for target 'venv/created' failed
make: *** [venv/created] Error 1

Alternatively, I backed up my OpenLane directory, then I used command git checkout master and then git pull, which works fine, but when I used the command make, I saw this error as below:

make[1]: Entering directory '/home/htf6ry/OpenLane'
57f3f21d6e1d0403a88d7a7973f7447597aae160: Pulling from efabless/openlane
Digest: sha256:0b9a27fe51d8d709e77f653c39dd1244738885d0d39fdefd81a272824a33b46f
Status: Image is up to date for efabless/openlane:57f3f21d6e1d0403a88d7a7973f7447597aae160
docker.io/efabless/openlane:57f3f21d6e1d0403a88d7a7973f7447597aae160
make[1]: Leaving directory '/home/htf6ry/OpenLane'
./venv/bin/python3 -m pip install --upgrade --no-cache-dir volare
/home/htf6ry/OpenLane/venv/bin/python3: No module named pip
Makefile:111: recipe for target 'pdk' failed
make: *** [pdk] Error 1

The problem is I do have pip and pip3 installed, the tool showed the error. I do not have any idea what happened....

AyaseErii commented 1 year ago

The pip issue has been solved since I happened to find changing include-system-site-packages = false to include-system-site-packages = true in OpenLane/venv/pyvenv.cfg before using make will solve the issue.

Now the issue is I cannot even pass the PDN step with the same design and same parameters (which you guys ran the flow and confirmed it can pass DRC). I put the issue_repo and error message here:

issue_reproducible.zip

[STEP 7]
[INFO]: Generating PDN (log: designs/fpga_core/runs/RUN_2022.08.18_16.14.21/logs/floorplan/7-pdn.log)...
[ERROR]: during executing openroad script /openlane/scripts/openroad/pdn.tcl
[ERROR]: Log: designs/fpga_core/runs/RUN_2022.08.18_16.14.21/logs/floorplan/7-pdn.log
[ERROR]: Last 10 lines:
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (190.400um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (204.000um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (217.600um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (176.800um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (190.400um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (204.000um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (217.600um, 73.440um), layer: 2.
[ERROR PSM-0069] Check connectivity failed.
Error: pdn.tcl, 51 PSM-0069
child process exited abnormally

The updated environment is:

python3 ./env.py issue-survey
Kernel: Linux v4.15.0-191-generic
Distribution: ubuntu 18.04
Python: v3.7.13 (OK)
Container Engine: docker v20.10.17 (OK)
OpenLane Git Version: 57f3f21d6e1d0403a88d7a7973f7447597aae160
pip: INSTALLED
python-venv: INSTALLED
---
PDK Version Verification Status: OK
---
Git Log (Last 3 Commits)

57f3f21 2022-08-18T13:38:05+02:00 Add Very Early support for GF180MCU + `open_pdks` Tweaks  (#1251) - Mohamed Gaber -  (HEAD -> master, origin/master, origin/HEAD)
c0be543 2022-08-16T16:47:08+02:00 Update OpenROAD (#1267) - Mohamed Gaber -  (tag: 2022.08.17)
7b15116 2022-08-15T18:27:17+02:00 Revamp Net Unbuffering (#1261) - Mohamed Gaber -  (tag: 2022.08.16)
---
Git Remotes

origin  https://github.com/The-OpenROAD-Project/OpenLane.git (fetch)
origin  https://github.com/The-OpenROAD-Project/OpenLane.git (push)

Any suggestions? Thanks!

maliberty commented 1 year ago

There is a known bug in check_connectivity which is currently being worked on. It is possible that this is a false positive though I haven't checked specifically. Do you have any macros in your design?

maliberty commented 1 year ago

@VidyaChhabria this may be the same as what you are looking at.

AyaseErii commented 1 year ago

@maliberty Thanks for the reply! Yes, I believe the design includes many macros. You may find the Github design repo here.

maliberty commented 1 year ago

Then I suspect this should wait for @VidyaChhabria fix. Until then I suggest you just turn off check_connectivity and rely on LVS to check.

VidyaChhabria commented 1 year ago

This should be resolved once this PR-2169 goes through and openroad is bumped up.

VidyaChhabria commented 1 year ago

Done.

This should be resolved once this PR-2169 goes through and openroad is bumped up.

AyaseErii commented 1 year ago

Done.

This should be resolved once this PR-2169 goes through and openroad is bumped up.

Thanks for the update! Should I update the OpenLane with the commands below?

    cd OpenLane/
    git checkout master
    git pull
    make 
    make test

AyaseErii commented 1 year ago

Hi, I did a quick test after using the commands for update, but it still shows the connectivity errors.... I attached the issue_repo and the error log below: issue_reproducible.zip

[STEP 7]
[INFO]: Generating PDN (log: designs/fpga_core/runs/RUN_2022.08.22_20.38.54/logs/floorplan/7-pdn.log)...
[ERROR]: during executing openroad script /openlane/scripts/openroad/pdn.tcl
[ERROR]: Log: designs/fpga_core/runs/RUN_2022.08.22_20.38.54/logs/floorplan/7-pdn.log
[ERROR]: Last 10 lines:
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (190.400um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (204.000um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (217.600um, 78.880um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (176.800um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (190.400um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (204.000um, 73.440um), layer: 2.
[WARNING PSM-0038] Unconnected PDN node on net VPWR at location (217.600um, 73.440um), layer: 2.
[ERROR PSM-0069] Check connectivity failed.
Error: pdn.tcl, 51 PSM-0069
child process exited abnormally

[ERROR]: Creating issue reproducible...

maliberty commented 1 year ago

A fix for the connectivity checker issue just went in today to OR.

donn commented 1 year ago

Should be okay now- feel free to re-open if not

The-OpenROAD-Project / OpenLane