LayoutViewer::drawRows() can be >100x slower than the rest of the rendering

[oharboe]

Describe the bug

LayoutViewer::drawRows() can be pathologically slow, which lead us down this wild goose chase in this issue ... Looking at the code, it looks like this method should render an approperiate level of detail to automatically balance speed and detail for zoom level.

This is odd. It could be a configuration issue, but I have tried a laptop and a very different workstation. Workstation is Ubuntu 22.04 and docker of course runs centos 7 inside the docker image.

Expected Behavior

Docker should be slower to zoom than a local install.

Environment

Git commit: aa1f8060d856733b82678802809c57bb260b3aac
kernel: Linux 5.19.0-41-generic
os: Ubuntu 22.10 (Kinetic Kudu)
cmake version 3.24.2

To Reproduce

Build locally ./build_openroad.sh --local and a docker image ./build_openroad.sh

Make a design that is large enough that zooming takes a noticable amount of time.

Create a "./or" script locally, then...

Runs in docker:

./or make gui_final

Runs locally:

make gui_final

zoom in/out rapidly, observe a difference in performance. I would not expect Docker to be faster.

#!/bin/bash
set -ex
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

WORKSPACE=$(pwd)

mkdir -p `pwd`/logs
mkdir -p `pwd`/results
mkdir -p `pwd`/fakeram-results
mkdir -p `pwd`/objects
mkdir -p `pwd`/reports

XSOCK=/tmp/.X11-unix
XAUTH=/tmp/.docker.xauth
xauth nlist :0 | sed -e 's/^..../ffff/' | xauth -f $XAUTH nmerge -
ARGUMENTS=$@

if test -t 0; then
    DOCKER_INTERACTIVE=-ti
fi

# Map the Verilog folder into the Docker image and build them
docker run -u $(id -u ${USER}):$(id -g ${USER}) \
 -e LIBGL_ALWAYS_SOFTWARE=1 \
 -e "QT_X11_NO_MITSHM=1" \
 -e XDG_RUNTIME_DIR=/tmp/xdg-run \
 -e DISPLAY=$DISPLAY \
 -e QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb \
 -v $XSOCK:$XSOCK \
 -v $XAUTH:$XAUTH \
 -e XAUTHORITY=$XAUTH \
 -e FLOW_HOME=/OpenROAD-flow-scripts/flow/ \
 -e MAKEFILES=/OpenROAD-flow-scripts/flow/Makefile \
 -v $WORKSPACE:`pwd` \
 $DOCKER_INTERACTIVE \
openroad/flow-centos7-builder:latest \
 bash -c "set -ex
 mkdir /tmp/xdg-run
 . ./env.sh
 cd $DIR
 $ARGUMENTS
"

Relevant log output

No response

Screenshots

No response

Additional Context

No response

[maliberty]

It sounds like this will be difficult to reproduce. Would you try adding

set_debug_level GUI draw 1

to your tcl file and report the results from both configurations. Perform the exact same operations in both runs (say initial load and three mouse wheel clicks close together).

[oharboe]

It sounds like this will be difficult to reproduce. Would you try adding

set_debug_level GUI draw 1

to your tcl file and report the results from both configurations. Perform the exact same operations in both runs (say initial load and three mouse wheel clicks close together).

I will make the measurement, but unless you can reproduce this on your end, then I don't think this finding will help us much...

Can you give it a go?

[oharboe]

@maliberty How do I measure GUI performance reliably?

Could a new target be made? make gui_final_test, load a design, then run performance tests?

[oharboe]

I didn't notice any additional output after setting up debug output. I ran the command then zoomed in/out:

[oharboe]

@nekronos Instructions on how to set this up in the debugger and using perf:

1. Build OpenROAD with debug information and release: ./build_openroad.sh --latest --local --openroad-args "-DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_CXX_FLAGS=-g"
2. Build a design that takes a bit of time to render and start GUI:

   cd flow
   MOCK_ARRAY_PITCH_SCALE=20 make DESIGN_CONFIG=designs/asap7/mock-array-big/config.mk
   make DESIGN_CONFIG=designs/asap7/mock-array-big/config.mk gui_final

3. Attach debugger
4. Zoom in/out and suspend/resume to examine where the time goes

Docker is faster.

If you want to profile using perf, you can run:

OPENROAD_EXE="perf record ../tools/install/OpenROAD/bin/openroad" make DESIGN_CONFIG=designs/asap7/mock-array-big/config.mk gui_final

Can we do the same with a software rasterizer using the local build?

sudo apt-get install libgl1-mesa-dri
export LIBGL_ALWAYS_SOFTWARE=1

I tried LIBGL_ALWAYS_SOFTWARE=1 on a laptop that clearly demonstrates slower local than docker, and I couldn't detect any effect.

[oharboe]

Nice! The difference evaporates on this laptop when I rebuild with the options above.

Docker, from zoom in, then preff Zoom in, then hit 'f', to render the full image:

Docker:

[DEBUG GUI-draw] inst search 0.00149054 sec
[DEBUG GUI-draw] inst outline render 7.8416e-05 sec
[DEBUG GUI-draw] layer V0 render 1.8518e-05 sec
[DEBUG GUI-draw] layer M1 render 0.0645053 sec
[DEBUG GUI-draw] layer V1 render 1.0153e-05 sec
[DEBUG GUI-draw] layer M2 render 0.0664876 sec
[DEBUG GUI-draw] layer V2 render 6.365e-06 sec
[DEBUG GUI-draw] layer M3 render 0.00801023 sec
[DEBUG GUI-draw] layer V3 render 1.1735e-05 sec
[DEBUG GUI-draw] layer M4 render 0.00827889 sec
[DEBUG GUI-draw] layer V4 render 1.085e-05 sec
[DEBUG GUI-draw] layer M5 render 0.00408528 sec
[DEBUG GUI-draw] layer V5 render 6.632e-06 sec
[DEBUG GUI-draw] layer M6 render 0.00302874 sec
[DEBUG GUI-draw] layer V6 render 6.802e-06 sec
[DEBUG GUI-draw] layer M7 render 8.445e-06 sec
[DEBUG GUI-draw] layer V7 render 5.346e-06 sec
[DEBUG GUI-draw] layer M8 render 8.196e-06 sec
[DEBUG GUI-draw] layer V8 render 5.12e-06 sec
[DEBUG GUI-draw] layer M9 render 1.0017e-05 sec
[DEBUG GUI-draw] layer V9 render 5.732e-06 sec
[DEBUG GUI-draw] layer Pad render 8.514e-06 sec
[DEBUG GUI-draw] total render 0.163417 sec

local OpenROAD v2.0-8120-gccbbca57b

[DEBUG GUI-draw] inst search 0.00151148 sec
[DEBUG GUI-draw] inst outline render 6.7555e-05 sec
[DEBUG GUI-draw] layer V0 render 8.846e-06 sec
[DEBUG GUI-draw] layer M1 render 0.0657221 sec
[DEBUG GUI-draw] layer V1 render 8.102e-06 sec
[DEBUG GUI-draw] layer M2 render 0.0666655 sec
[DEBUG GUI-draw] layer V2 render 5.624e-06 sec
[DEBUG GUI-draw] layer M3 render 0.00730094 sec
[DEBUG GUI-draw] layer V3 render 6.418e-06 sec
[DEBUG GUI-draw] layer M4 render 0.00745809 sec
[DEBUG GUI-draw] layer V4 render 6.413e-06 sec
[DEBUG GUI-draw] layer M5 render 0.00412683 sec
[DEBUG GUI-draw] layer V5 render 5.133e-06 sec
[DEBUG GUI-draw] layer M6 render 0.00311936 sec
[DEBUG GUI-draw] layer V6 render 4.067e-06 sec
[DEBUG GUI-draw] layer M7 render 5.354e-06 sec
[DEBUG GUI-draw] layer V7 render 2.76e-06 sec
[DEBUG GUI-draw] layer M8 render 3.785e-06 sec
[DEBUG GUI-draw] layer V8 render 2.794e-06 sec
[DEBUG GUI-draw] layer M9 render 3.661e-06 sec
[DEBUG GUI-draw] layer V9 render 2.792e-06 sec
[DEBUG GUI-draw] layer Pad render 3.658e-06 sec
[DEBUG GUI-draw] total render 0.161982 sec

<!DOCTYPE html>

  |   | docker | local | difference in percent of total running time -- | -- | -- | -- | -- inst | search | 1.49E-03 | 1.51E-03 | -0.01% inst | outline | 7.84E-05 | 6.76E-05 | 0.01% layer | V0 | 1.85E-05 | 8.85E-06 | 0.01% layer | M1 | 6.45E-02 | 6.57E-02 | -0.74% layer | V1 | 1.02E-05 | 8.10E-06 | 0.00% layer | M2 | 6.65E-02 | 6.67E-02 | -0.11% layer | V2 | 6.37E-06 | 5.62E-06 | 0.00% layer | M3 | 8.01E-03 | 7.30E-03 | 0.43% layer | V3 | 1.17E-05 | 6.42E-06 | 0.00% layer | M4 | 8.28E-03 | 7.46E-03 | 0.50% layer | V4 | 1.09E-05 | 6.41E-06 | 0.00% layer | M5 | 4.09E-03 | 4.13E-03 | -0.03% layer | V5 | 6.63E-06 | 5.13E-06 | 0.00% layer | M6 | 3.03E-03 | 3.12E-03 | -0.06% layer | V6 | 6.80E-06 | 4.07E-06 | 0.00% layer | M7 | 8.45E-06 | 5.35E-06 | 0.00% layer | V7 | 5.35E-06 | 2.76E-06 | 0.00% layer | M8 | 8.20E-06 | 3.79E-06 | 0.00% layer | V8 | 5.12E-06 | 2.79E-06 | 0.00% layer | M9 | 1.00E-05 | 3.66E-06 | 0.00% layer | V9 | 5.73E-06 | 2.79E-06 | 0.00% layer | Pad | 8.51E-06 | 3.66E-06 | 0.00% total |   | 1.63E-01 | 1.62E-01 | 0.88%

[oharboe]

@maliberty On @nerriav laptop it is still slow. Need further investigation.

[oharboe]

@nerriav Please report by measuring as follows:

1. start GUI
2. zoom in
3. type set_debug_level GUI draw 1 into the OpenROAD GUI prompt
4. press 'f'
5. Copy and paste the measurements here for Docker & local, like I did here: https://github.com/The-OpenROAD-Project/OpenROAD/issues/3330#issuecomment-1547600882

[nerriav]

Hi @oharboe Table attached. calculate differences between the platform as follow : diffin% = (local - docker) / docker *100 See that it took for local - ORFS 35 seconds !!!		docker	local	difference in percent of total running time
inst	search	0.000955899	1.06E-02	100641.40%
inst	outline	6.91E-05	2.58E-02	3720648.50%
layer	V0	9.47E-06	8.22E-06	-1315.23%
layer	M1	0.0645423	1.19E-01	8376.01%
layer	V1	1.20E-05	4.35E-06	-6367.97%
layer	M2	0.0639514	1.20E-01	8716.87%
layer	V2	1.00E-05	5.57E-06	-4436.68%
layer	M3	0.00816242	1.11E-02	3597.19%
layer	V3	3.43E-05	3.86E-06	-8872.41%
layer	M4	0.00845554	1.17E-02	3797.11%
layer	V4	1.14E-05	3.60E-06	-6841.09%
layer	M5	0.00406955	1.10E-02	16997.83%
layer	V5	5.82E-06	4.87E-06	-1638.05%
layer	M6	0.00314803	8.57E-03	17227.22%
layer	V6	5.60E-06	3.37E-06	-3979.63%
layer	M7	4.02E-05	2.43E-03	593041.45%
layer	V7	5.45E-06	2.47E-06	-5472.91%
layer	M8	2.47E-05	1.97E-03	789858.04%
layer	V8	2.20E-05	2.56E-06	-8833.84%
layer	M9	2.42E-05	1.89E-03	768996.66%
layer	V9	6.33E-06	2.81E-06	-5557.31%
layer	Pad	1.61E-05	1.83E-03	1123897.46%
total		0.161439	3.53E+01	2175847.29%

[maliberty]

The local time is odd as the components don't add up to anywhere near the total. I don't print every bit of the rendering, just what are normally the key parts. It seems something else is taking surprisingly long time.

Do you have time to throw in some more timers around the other steps and try to narrow down where the extra time is going?

The is all rather mysterious but quite interesting.

[oharboe]

@nerriav Also tried deleting the entire OpenROAD-flow-scripts folder and rebuilding, same result.

[oharboe]

The rendering speed in the accounted for part of the rendering is ca. 2x the accounted for time in local build. The accounted for time is 1% of the total time in the local build.

So there are two findings here: docker rendering is 2x slower and there is a large unaccounted for time sink. Both should be fixed, but perhaps they share a cause?

<!DOCTYPE html>

accounted for above |   | 1.54E-01 | 3.26E-01 -- | -- | -- | --

[oharboe]

@nerriav Try again and report the numbers. I added some extra timers. Near as I can tell we should now account for all the time spent in drawing, so the numbers should add up. If not, the next step is to run this in the debugger: https://github.com/The-OpenROAD-Project/OpenROAD/pull/3340

To build this PR locally, do the following:

cd OpenROAD-flow-scripts/tools/OpenROAD
git fetch
git checkout origin/master
git fetch https://github.com/Pinata-Consulting/OpenROAD.git gui-more-timers
git merge FETCH_HEAD
cd ../..
git add tools/OpenROAD
git commit -m blah
./build_openroad.sh --local --openroad-args "-D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_CXX_FLAGS=-g"

[nerriav]

manufacturing grid 0.000121689 sec inst search 0.0147962 sec inst outline render 0.0407303 sec blockages 4.598e-06 sec layer V0 render 9.077e-06 sec layer M1 render 0.202713 sec layer V1 render 9.471e-06 sec layer M2 render 0.207613 sec layer V2 render 1.6505e-05 sec layer M3 render 0.0147582 sec layer V3 render 9.468e-06 sec layer M4 render 0.0142019 sec layer V4 render 8.315e-06 sec layer M5 render 0.0159621 sec layer V5 render 6.121e-06 sec layer M6 render 0.0149414 sec layer V6 render 7.889e-06 sec layer M7 render 0.00435427 sec layer V7 render 5.366e-06 sec layer M8 render 0.00368478 sec layer V8 render 5.972e-06 sec layer M9 render 0.00352584 sec layer V9 render 6.7e-06 sec layer Pad render 0.00312212 sec various 54.8853 sec total render 55.4305 sec

[nerriav]

@oharboe second update

manufacturing grid 0.000166429 sec inst search 0.0139036 sec inst outline render 0.0394747 sec blockages 7.365e-06 sec layer V0 render 1.1001e-05 sec layer M1 render 0.191828 sec layer V1 render 9.09e-06 sec layer M2 render 0.201626 sec layer V2 render 2.0946e-05 sec layer M3 render 0.0147654 sec layer V3 render 6.435e-06 sec layer M4 render 0.015414 sec layer V4 render 6.146e-06 sec layer M5 render 0.0150112 sec layer V5 render 7.758e-06 sec layer M6 render 0.0133223 sec layer V6 render 7.144e-06 sec layer M7 render 0.00406165 sec layer V7 render 6.511e-06 sec layer M8 render 0.00360056 sec layer V8 render 5.969e-06 sec layer M9 render 0.0032807 sec layer V9 render 6.581e-06 sec layer Pad render 0.00321918 sec instance names 0.00387538 sec rows 53.9299 sec access points 7.68e-06 sec module view 0.00140507 sec regions 7.799e-06 sec pin markers 2.221e-06 sec save cell grid 2.312e-06 sec save restore 0.000163636 sec total render 54.4609 sec

[nerriav]

@oharboe third update

manufacturing grid 0.00020926 sec inst search 0.0132036 sec inst outline render 0.0426823 sec blockages 6.178e-06 sec layer V0 render 1.0863e-05 sec layer M1 render 0.192389 sec layer V1 render 1.1728e-05 sec layer M2 render 0.188142 sec layer V2 render 9.735e-06 sec layer M3 render 0.0149424 sec layer V3 render 6.764e-06 sec layer M4 render 0.0148525 sec layer V4 render 6.811e-06 sec layer M5 render 0.018877 sec layer V5 render 6.1e-06 sec layer M6 render 0.015878 sec layer V6 render 6.834e-06 sec layer M7 render 0.0045793 sec layer V7 render 4.798e-06 sec layer M8 render 0.00384125 sec layer V8 render 4.26e-06 sec layer M9 render 0.00305581 sec layer V9 render 3.411e-06 sec layer Pad render 0.00319053 sec instance names 0.00372407 sec rows 58.4752 sec access points 7.32e-06 sec module view 0.00184812 sec regions 8.743e-06 sec pin markers 2.671e-06 sec save cell grid 2.638e-06 sec save restore 0.00018609 sec total render 59.0027 sec

[nerriav]

much better !

manufacturing grid 0.000214012 sec inst search 0.0148889 sec inst outline render 0.0400965 sec blockages 5.37e-06 sec layer V0 render 1.0145e-05 sec layer M1 render 0.191335 sec layer V1 render 1.084e-05 sec layer M2 render 0.202215 sec layer V2 render 1.0356e-05 sec layer M3 render 0.0135547 sec layer V3 render 9.821e-06 sec layer M4 render 0.013858 sec layer V4 render 8.187e-06 sec layer M5 render 0.0148435 sec layer V5 render 8.326e-06 sec layer M6 render 0.0122909 sec layer V6 render 7.354e-06 sec layer M7 render 0.00412806 sec layer V7 render 5.03e-06 sec layer M8 render 0.00353591 sec layer V8 render 6.205e-06 sec layer M9 render 0.00314508 sec layer V9 render 6.324e-06 sec layer Pad render 0.00441037 sec instance names 0.00492425 sec rows 4.17739 sec access points 7.483e-06 sec module view 0.00148594 sec regions 5.792e-06 sec pin markers 1.763e-06 sec save cell grid 1.838e-06 sec save restore 0.00017311 sec total render 4.70823 sec

[oharboe]

@nerriav Try mv ~/.config/OpenRoad\ Project/openroad.conf ~/.config/OpenRoad\ Project/openroad.backup

[nerriav]

[DEBUG GUI-draw] total render 0.312251 sec issue solved :1st_place_medal: Many thanks

[oharboe]

@maliberty @nerriav The difference between docker and local was the configuration file.

I thought the problem was with rows being rendred, but I need to dive into the config.zip file that Neria will upload.

[nerriav]

config.zip

[oharboe]

config.zip

I tried mv openroad.backup ~/.config/OpenRoad\ Project/openroad.conf, but didn't observe any slowdown.

[oharboe]

@maliberty Although we don't know exactly what is going on, we have some more evidence... Can you have a look and see if you can guess what is going on?

Branch with work in progress, I'm uncertain about how and whether, more timers to the gui should be added after these findings: https://github.com/Pinata-Consulting/OpenROAD/tree/gui-more-timers

[maliberty]

Do you have Detailed view on or off when meauring the row performance?

[maliberty]

With rows turned on you will see a lot of green boxes. Did you not observe a large difference in the layout appearance between docker and non-docker runs?

[nerriav]

with both ROWs and 'detailed view; turned on, got the following results : Docker : total render 63.9329 sec Local : total render 63.1835 sec whereas previous checks which I tested on floorplan GUI, this one I did with place GUI

[oharboe]

with both ROWs and 'detailed view; turned on, got the following results : Docker : total render 63.9329 sec Local : total render 63.1835 sec whereas previous checks which I tested on floorplan GUI, this one I did with place GUI

Super!

This dispels of the mysteries about the slowdown....

After all this, I'm thinking it would be great with a "reset settings to default" button in the GUI... Could help reduce confusion during testing reporting?

We've had a conversation about GUI setting several times...

[maliberty]

Rows will be the same in floorplan or place so that shouldn't matter. We have -no_init we could also have a -no_settings flag to OR.

[oharboe]

Rows will be the same in floorplan or place so that shouldn't matter. We have -no_init we could also have a -no_settings flag to OR.

I would have thought the settings file was a -gui specific thing, so no need for a new option: dont read in when -gui is not specifi

[oharboe]

Rows will be the same in floorplan or place so that shouldn't matter. We have -no_init we could also have a -no_settings flag to OR.

I would have thought the settings file was a -gui specific thing, so no need for a new option: dont read in when -gui is not specified.

I was thinking about a "reset to defaults" GUI button.

[maliberty]

If -gui isn't specified then what rendering problem are we discussing?

We do have the offscreen rendering during ORFS to save images. The settings would affect such operations though we reset all the values first

[oharboe]

If -gui isn't specified then what rendering problem are we discussing?

We do have the offscreen rendering during ORFS to save images. The settings would affect such operations though we reset all the values first

What you said abovem

[maliberty]

"What you said abovem" I don't know what you are referring to.

[oharboe]

Answering too quickly on phone ... trying again:

I thought you suggested a -no_settings for the GUI. For the GUI case, I think a reset settings button would make more sense. For the non -gui case, it sounds like the settings are not used today. So, I dont see the usecase for a -no_settings option.

[maliberty]

The performance issue is addressed in #3342

Yes the -no_setting is for the GUI. I'm not sure how the reset button "Could help reduce confusion during testing reporting?". I would have to ask if someone used it or not and if they made any changes afterwards. It is simpler to just say run -no_settings. There could be a GUI_NO_SETTINGS control on the flow.

[maliberty]

Fwiw I have this in my ~/.openroad which gives me the reset to exactly the settings I prefer:

  create_toolbar_button -text "Reset" -script {
    gui::set_display_controls "Layers/*" visible true
    gui::set_display_controls "Nets/*" visible true
    gui::set_display_controls "Instances/*" visible true
    gui::set_display_controls "Blockages/*" visible true
    gui::set_display_controls "Rulers" visible true
    gui::set_display_controls "Rows/*" visible false
    gui::set_display_controls "Pin Markers" visible true
    gui::set_display_controls "Tracks/*" visible false
    gui::set_display_controls "Misc/*" visible false
    gui::set_display_controls "Misc/Scale bar" visible true
    gui::set_display_controls "Misc/Detailed view" visible true
    gui::set_display_controls "Misc/Highlight selected" visible true
    gui::set_display_controls "Misc/Instances/Pins" visible true
    gui::set_display_controls "Misc/Instances/Blockages" visible true
    gui::set_display_controls "Heat Maps/*" visible false
  }

[oharboe]

The performance issue is addressed in #3342

Yes the -no_setting is for the GUI. I'm not sure how the reset button "Could help reduce confusion during testing reporting?". I would have to ask if someone used it or not and if they made any changes afterwards. It is simpler to just say run -no_settings. There could be a GUI_NO_SETTINGS control on the flow.

I think that saving setting is of dubious value to begin with as one switches between designs and final to floorplan. it has consistently caused us more problems than helping us. one advantage of the docker image is that it starts wo a settings file. A reset to defaults would be independently useful. I doubt that anyone will remember to add the -no_settings option.

I can easily emulate -no_settings by erasing the settings config file.

[maliberty]

I find the saving of the window locations and sizes very helpful and wouldn't want to have to do that each time. The display controls are a bit less important and I could live with them being reset each time. I'd like to hear from other users on their feelings (@gadfort @rovinski @QuantamHD )

[oharboe]

I find the saving of the window locations and sizes very helpful and wouldn't want to have to do that each time. The display controls are a bit less important and I could live with them being reset each time. I'd like to hear from other users on their feelings (@gadfort @rovinski @QuantamHD )

👍 I think if OpenROAD has a robust balance between detail and rendering speed(it is definitely getting there!), then I would be less concerned about saving settings or not. For novice users a "reset to defaults" button clearly visible, I think would be helpful as well as useful for debugging.

[gadfort]

@maliberty I like to have the settings as well, its rare that I want to reset everyone wholesale, unless I've corrupted the GUI settings somehow. (I suppose we could lean on the -no_init flag to determine if the we should load the settings). What happens when the GUI is relaunched? back to normal behavior?

@oharboe we don't really store the defaults anywhere, so that would require a little more effort to implement, but could be a useful feature if others need it.

[oharboe]

@maliberty The problem of this issue is solved. There are some followup discussions taking place here, which is fine I guess.

Close issue?

[rovinski]

Personally I like having the settings saved. If I need to "reset" then I would just turn off all visibility and then only turn on the ones I want.