Benchmark reference sizing results difference

[Akilax0]

Currently using the provided evaluation.py to test the given .size output files of the benchmark. However, I keep getting different values than the provided benchmark results at /design/README.md

For example running

would give the output as,

while the provided outputs are

[bingyuew]

Please confirm you are using the updated benchmark.

[Akilax0]

Pulled again just now. Still the same results.

[bingyuew]

Please provide us with the benchmark you are using to reproduce the issue, including all necessary scripts.

[Akilax0]

I have direct cloned the repository. https://github.com/ASU-VDA-Lab/2024_ICCAD_Contest_Gate_Sizing_Benchmark/tree/main

unzipped benchmark using the given script as unzip.sh

The script is unchanged : evaluation.py

for file_path : ../../design/NV_NVDLA_partition_m/NV_NVDLA_partition_m.size for design_name : NV_NVDLA_partition_m

(as in the first screenshot )

[bingyuew]

Please provide the files you are currently using so we can reproduce the issue.

[Akilax0]

ICCAD_issue.zip

README.md has the instructions to reproduce

[bingyuew]

I cannot reproduce your issue. Please rebuild the OpenROAD binary and confirm that you are using the most up-to-date version of OpenROAD.

[Akilax0]

I have rebuild and tested the same files on two separate machines. I keep getting the same output as above.

The outputs you got for the provided files , are they the same as the given results at /design/README.md ?

[bingyuew]

Yes, I am getting the same result as in the README file. Please make sure you have updated the submodules.

[vickychen928]

@Akilax0 I encountered the same error as well. Have you managed to resolve it?

[bingyuew]

@vickychen928 @Akilax0 Could both of you share the message from your cmake and the process of building the openroad binary?

[vickychen928]

Here is the cmake and the process of building the openroad binary: cmake_log.txt make_log.txt

[Akilax0]

Yeah, I get the same values still. I also tried with the docker file same result. Here are my cmake and make outputs.

make_log.txt cmake_log.txt

[bingyuew]

Hi @Akilax0 and @vickychen928 , please try using this Dockerfile.txt and let me know if the problem is fixed.

[pipiofzl]

I encountered the same issue when running the provided example with OpenROAD_example.py. The output.size file generated was then used in the evaluation script evaluation.py, which produced the results. Similarly, when I used the design/NV_NVDLA_partition_m/NV_NVDLA_partition_m.size file with the evaluation.py script, the results did not match the Statistics provided on the GitHub page for NV_NVDLA_partition_m. I would like to ask which programs were used to generate the Benchmark Statistics and the Reference Sizing Result Statistics provided on GitHub.

Thank you.

[bingyuew]

@pipiofzl Can you also share the message from your cmake and the process of building the openroad binary?

[vickychen928]

@bingyuew I would like to ask if I need to make changes because the version specified in docker.txt is Ubuntu 22.04, but the computing platform I am using is Ubuntu 20.04?

[pipiofzl]

Here is the cmake and the process of building the openroad binary: cmake .. instruction: cmake.log make j 6 instruction: make_j_6.log make install instruction: make_install.log

Thank you.

[Akilax0]

@bingyuew I got the same results as @pipiofzl when using the Dockerfile

[Akilax0]

@vickychen928 Shouldn't be an issue. Any reason why you asked this?

[vickychen928]

@Akilax0 There's no particular reason, just concerned whether specifying a different version from the platform might cause errors.