search

The-OpenROAD-Project / OpenLane

OpenLane is an automated RTL to GDSII flow based on several components including OpenROAD, Yosys, Magic, Netgen and custom methodology scripts for design exploration and optimization.
https://openlane.readthedocs.io/
Apache License 2.0
1.25k stars 365 forks source link

Add python `libparse` dependency to `requirements.txt` #2107

Closed dlmiles closed 2 months ago

dlmiles commented 2 months ago

Description

$ find . -type f -name "*.py" -exec fgrep -Hn libparse {} \; | grep import
./scripts/utils/get_default_operating_conditions.py:18:from libparse import LibertyParser

Please add this library to the appropriate requirements.txt file, I have seen it needed at runtime when using the flow to generate GDS.

0.3.1 appears to be the version I need to manually install so I do not see a late stage flow failure.

libparse>=0.3.1

Expected Behavior

I do not expect the flow runtime to fail in the later stages with missing Python module errors concerning libparse because it was not installed.

Environment report

All OL tags for the past 6+ months appear to be affected.

Reproduction material

All OL tags for the past 6+ months appear to be affected.

Relevant log output

Not to hand, but an audit of the `*.py` and the `import` declarations should be enough to see the dependency exists.
donn commented 2 months ago

Thanks for catching that. I'm on it.

donn commented 2 months ago

So-- this is something of an idiosyncrasy of OpenLane 1's but long story short, requirements.txt are primarily concerned with people installing OpenLane, not using it. The runtime requirements are in dependencies/python/run_time.txt.

I'll go ahead and update it while I'm at it. Sorry for the confusion.

dlmiles commented 2 months ago

Sure I did see there were 5 requirements.txt like files in the tree. I use the term in the generic Python sense of the mechanism.

Hence my use of the term Please add this library to the appropriate requirements.txt file as I do not know the individual details of all 5.

dlmiles commented 2 months ago

This is how things look to me without the change.

$ pip list
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Package            Version
------------------ -------
click              8.0.4
importlib-metadata 4.8.3
libparse           0.1.5
pip                21.3.1
PyYAML             6.0.1
setuptools         39.2.0
typing_extensions  4.1.1
wheel              0.37.1
XlsxWriter         3.1.2
zipp               3.6.0

$ ls -l venv
ls: cannot access venv: No such file or directory

The setup sequence:

OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ prepare
Cloning into '/data/vlsi/openlane_2024.04.02'...
remote: Enumerating objects: 508, done.
remote: Counting objects: 100% (508/508), done.
remote: Compressing objects: 100% (431/431), done.
remote: Total 508 (delta 77), reused 311 (delta 32), pack-reused 0
Receiving objects: 100% (508/508), 8.33 MiB | 31.59 MiB/s, done.
Resolving deltas: 100% (77/77), done.
Note: switching to 'eaba5192c45aa333ab45216ce1773d75d539e9b3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

###
### docker info --format {{json .}}
###
###
### docker info --format {{json .}}
###
make[1]: Entering directory `/data/vlsi/openlane_2024.04.02'
###
### docker pull efabless/openlane:eaba5192c45aa333ab45216ce1773d75d539e9b3
###
make[1]: Leaving directory `/data/vlsi/openlane_2024.04.02'
rm -rf ./venv
python3 -m venv ./venv
PYTHONPATH= ./venv/bin/python3 -m pip install --upgrade --no-cache-dir pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)
    100% |████████████████████████████████| 1.7MB 51.9MB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-21.3.1
You are using pip version 21.3.1, however version 24.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
PYTHONPATH= ./venv/bin/python3 -m pip install --upgrade --no-cache-dir -r ./requirements_dev.txt
Collecting cmake
  Downloading cmake-3.28.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.3 MB)
     |████████████████████████████████| 26.3 MB 4.3 MB/s
Requirement already satisfied: pip in ./venv/lib/python3.6/site-packages (from -r ./././dependencies/python/precompile_time.txt (line 1)) (21.3.1)
Collecting wheel
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting click<9,>=8.0.0
  Downloading click-8.0.4-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 29.1 MB/s
Collecting pyyaml<7,>=6
  Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)
     |████████████████████████████████| 677 kB 27.6 MB/s
Collecting black~=22.3.0
  Downloading black-22.3.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 48.0 MB/s
Collecting flake8~=4.0.1
  Downloading flake8-4.0.1-py2.py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 13.5 MB/s
Collecting flake8-no-implicit-concat==0.3.3
  Downloading flake8_no_implicit_concat-0.3.3-py3-none-any.whl (5.2 kB)
Collecting more-itertools
  Downloading more_itertools-8.14.0-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 10.4 MB/s
Collecting importlib-metadata
  Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB)
Collecting tomli>=1.1.0
  Downloading tomli-1.2.3-py3-none-any.whl (12 kB)
Collecting dataclasses>=0.6
  Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
Collecting pathspec>=0.9.0
  Downloading pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting typing-extensions>=3.10.0.0
  Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Collecting mypy-extensions>=0.4.3
  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting platformdirs>=2
  Downloading platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting typed-ast>=1.4.2
  Downloading typed_ast-1.5.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (777 kB)
     |████████████████████████████████| 777 kB 59.0 MB/s
Collecting mccabe<0.7.0,>=0.6.0
  Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.5.0,>=2.4.0
  Downloading pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 18.7 MB/s
Collecting pycodestyle<2.9.0,>=2.8.0
  Downloading pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 26.0 MB/s
Collecting importlib-metadata
  Downloading importlib_metadata-4.2.0-py3-none-any.whl (16 kB)
Collecting zipp>=0.5
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Installing collected packages: zipp, typing-extensions, pyflakes, pycodestyle, mccabe, importlib-metadata, typed-ast, tomli, platformdirs, pathspec, mypy-extensions, more-itertools, flake8,
Successfully installed black-22.3.0 click-8.0.4 cmake-3.28.4 dataclasses-0.8 flake8-4.0.1 flake8-no-implicit-concat-0.3.3 importlib-metadata-4.2.0 mccabe-0.6.1 more-itertools-8.14.0 mypy-ex
touch venv/created
PYTHONPATH= ./venv/bin/python3 -m pip install --upgrade --no-cache-dir 'volare>=0.12.3'
Collecting volare>=0.12.3
  Downloading volare-0.16.0-py3-none-any.whl (35 kB)
Collecting pcpp<2,>=1.2
  Downloading pcpp-1.30-py2.py3-none-any.whl (91 kB)
     |████████████████████████████████| 91 kB 57.0 MB/s
Requirement already satisfied: pyyaml<7,>=5 in ./venv/lib/python3.6/site-packages (from volare>=0.12.3) (6.0.1)
Collecting httpx>=0.22.0
  Downloading httpx-0.22.0-py3-none-any.whl (84 kB)
     |████████████████████████████████| 84 kB 7.6 MB/s
Collecting rich<14,>=12
  Downloading rich-12.6.0-py3-none-any.whl (237 kB)
     |████████████████████████████████| 237 kB 56.7 MB/s
Collecting zstandard<1,>=0.19.0
  Downloading zstandard-0.20.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)
     |████████████████████████████████| 2.6 MB 26.2 MB/s
Requirement already satisfied: click<9,>=8.0.0 in ./venv/lib/python3.6/site-packages (from volare>=0.12.3) (8.0.4)
Requirement already satisfied: importlib-metadata in ./venv/lib/python3.6/site-packages (from click<9,>=8.0.0->volare>=0.12.3) (4.2.0)
Collecting sniffio
  Downloading sniffio-1.2.0-py3-none-any.whl (10 kB)
Collecting certifi
  Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
     |████████████████████████████████| 163 kB 32.8 MB/s
Collecting async-generator
  Downloading async_generator-1.10-py3-none-any.whl (18 kB)
Collecting httpcore<0.15.0,>=0.14.5
  Downloading httpcore-0.14.7-py3-none-any.whl (68 kB)
     |████████████████████████████████| 68 kB 12.7 MB/s
Collecting rfc3986[idna2008]<2,>=1.3
  Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting charset-normalizer
  Downloading charset_normalizer-3.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (162 kB)
     |████████████████████████████████| 162 kB 2.0 MB/s
Collecting commonmark<0.10.0,>=0.9.0
  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 2.1 MB/s
Requirement already satisfied: dataclasses<0.9,>=0.7 in ./venv/lib/python3.6/site-packages (from rich<14,>=12->volare>=0.12.3) (0.8)
Collecting pygments<3.0.0,>=2.6.0
  Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 32.8 MB/s
Requirement already satisfied: typing-extensions<5.0,>=4.0.0 in ./venv/lib/python3.6/site-packages (from rich<14,>=12->volare>=0.12.3) (4.1.1)
Collecting h11<0.13,>=0.11
  Downloading h11-0.12.0-py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 15.4 MB/s
Collecting anyio==3.*
  Downloading anyio-3.6.2-py3-none-any.whl (80 kB)
     |████████████████████████████████| 80 kB 30.0 MB/s
Collecting contextvars>=2.1
  Downloading contextvars-2.4.tar.gz (9.6 kB)
  Preparing metadata (setup.py) ... done
Collecting idna>=2.8
  Downloading idna-3.6-py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 4.6 MB/s
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.6/site-packages (from importlib-metadata->click<9,>=8.0.0->volare>=0.12.3) (3.6.0)
Collecting immutables>=0.9
  Downloading immutables-0.19-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (116 kB)
     |████████████████████████████████| 116 kB 35.0 MB/s
Building wheels for collected packages: contextvars
  Building wheel for contextvars (setup.py) ... done
  Created wheel for contextvars: filename=contextvars-2.4-py3-none-any.whl size=7665 sha256=69a0e58c6cb5edfcba1c16935bd2510288034b8a8223797dcfc3a8b9d60b9278
  Stored in directory: /tmp/pip-ephem-wheel-cache-4me5s5nw/wheels/41/11/53/911724983aa48deb94792432e14e518447212dd6c5477d49d3
Successfully built contextvars
Installing collected packages: immutables, contextvars, sniffio, idna, rfc3986, h11, certifi, anyio, pygments, httpcore, commonmark, charset-normalizer, async-generator, zstandard, rich, pc
Successfully installed anyio-3.6.2 async-generator-1.10 certifi-2024.2.2 charset-normalizer-3.0.1 commonmark-0.9.1 contextvars-2.4 h11-0.12.0 httpcore-0.14.7 httpx-0.22.0 idna-3.6 immutable
./venv/bin/volare enable --pdk sky130
Version cd1748bb197f9b7af62a54507de6624e30363943 enabled for the sky130 PDK.
Version 6d4d11780c40b20ee63cc98e645307a9bf2b2ab8 enabled for the sky130 PDK.

pip list | grep libparse
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
libparse           0.1.5

### docker commands:
docker info --format {{json .}}
docker info --format {{json .}}
docker pull efabless/openlane:eaba5192c45aa333ab45216ce1773d75d539e9b3

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ echo PDK=$PDK
PDK=sky130A
[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ echo PDK_VERSION=$PDK_VERSION
PDK_VERSION=6d4d11780c40b20ee63cc98e645307a9bf2b2ab8
[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ openroad -version
0889970d1790a2617e69f253221b8bd7626e51dc 
[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ echo $OPENLANE_TAG
2024.04.02
[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ echo OPENLANE_TAG=$OPENLANE_TAG
OPENLANE_TAG=2024.04.02

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ which pip
/usr/local/bin/pip

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ pip list
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
Package            Version
------------------ -------
click              8.0.4
importlib-metadata 4.8.3
libparse           0.1.5
pip                21.3.1
PyYAML             6.0.1
setuptools         39.2.0
typing_extensions  4.1.1
wheel              0.37.1
XlsxWriter         3.1.2
zipp               3.6.0

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ pip show libparse
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
Name: libparse
Version: 0.1.5
Summary: Python wrapper around Yosys' libparse module
Home-page: UNKNOWN
Author: Efabless Corporation and Contributors
Author-email: donn@efabless.com
License: UNKNOWN
Location: /usr/local/lib64/python3.6/site-packages
Requires: wheel
Required-by: 

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ ls -ald venv/
drwxr-xr-x 5 dlm dlm 4096 Apr  3 21:42 venv/

[OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ source venv/bin/activate

(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ which pip
/data/vlsi/openlane_2024.04.02/venv/bin/pip

(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ pip list
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
Package                   Version
------------------------- --------
anyio                     3.6.2
async-generator           1.10
black                     22.3.0
certifi                   2024.2.2
charset-normalizer        3.0.1
click                     8.0.4
cmake                     3.28.4
commonmark                0.9.1
contextvars               2.4
dataclasses               0.8
flake8                    4.0.1
flake8-no-implicit-concat 0.3.3
h11                       0.12.0
httpcore                  0.14.7
httpx                     0.22.0
idna                      3.6
immutables                0.19
importlib-metadata        4.2.0
mccabe                    0.6.1
more-itertools            8.14.0
mypy-extensions           1.0.0
pathspec                  0.9.0
pcpp                      1.30
pip                       21.3.1
platformdirs              2.4.0
pycodestyle               2.8.0
pyflakes                  2.4.0
Pygments                  2.14.0
PyYAML                    6.0.1
rfc3986                   1.5.0
rich                      12.6.0
setuptools                39.2.0
sniffio                   1.2.0
tomli                     1.2.3
typed-ast                 1.5.5
typing_extensions         4.1.1
volare                    0.16.0
wheel                     0.37.1
zipp                      3.6.0
zstandard                 0.20.0

(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ pip show libparse
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
WARNING: Package(s) not found: libparse

Then I manually run:

(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ ./venv/bin/python3 -m pip install libparse
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of th
Collecting libparse
  Downloading libparse-0.3.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 15.6 MB/s
Requirement already satisfied: wheel in ./venv/lib/python3.6/site-packages (from libparse) (0.37.1)
Installing collected packages: libparse
Successfully installed libparse-0.3.1
(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@host360B /data/vlsi/openlane_2024.04.02]$ 

(venv) [OL:2024.04.02 OR:0889970d1790a2617e69f253221b8bd7626e51dc dlm@363C00B /data/vlsi/openlane_2024.04.02]$ pip show libparse
WARNING: The directory '/openlane/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Name: libparse
Version: 0.3.1
Summary: Python wrapper around Yosys' libparse module
Home-page: UNKNOWN
Author: Efabless Corporation and Contributors
Author-email: donn@efabless.com
License: UNKNOWN
Location: /data/vlsi/openlane_2024.04.02/venv/lib/python3.6/site-packages
Requires: wheel
Required-by:

This is how the flow looks when it runs and has the error (outside env has libparse 0.1.5 installed, inside env does not have any libparse installed - at time it errors):

[STEP 31]
[INFO]: Creating IR Drop Report (log: ../work/runs/wokwi/logs/signoff/31-irdrop.log)...
[ERROR]: during executing openroad script /openlane/scripts/openroad/irdrop.tcl
[ERROR]: Log: ../work/runs/wokwi/logs/signoff/31-irdrop.log
[ERROR]: Last 10 lines:
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[INFO]: Reading ODB at '/work/runs/wokwi/results/routing/tt_um_no_time_for_squares_tommythorn.odb'…
define_corners Typical
read_liberty -corner Typical /data/PDK_ROOT/sky130A/libs.ref/sky130_fd_sc_hd/lib/sky130_fd_sc_hd__tt_025C_1v80.lib
Error: irdrop.tcl, 15 Traceback (most recent call last):
  File "/openlane/scripts/utils/get_default_operating_conditions.py", line 18, in <module>
    from libparse import LibertyParser
ModuleNotFoundError: No module named 'libparse'
child process exited abnormally

[ERROR]: Creating issue reproducible...
donn commented 2 months ago

Addressed with #2108.