search

The-OpenROAD-Project / megaboom

BSD 3-Clause "New" or "Revised" License
15 stars 3 forks source link

Jenkins reports SUCCESS when the build actually failed #122

Open jeffng-or opened 3 hours ago

jeffng-or commented 3 hours ago

Found in https://github.com/The-OpenROAD-Project/megaboom/pull/120. The bazel build failed and reported errors, but Jenkins said that the job was a success.

jeffng-or commented 2 hours ago

@maliberty @oharboe - I've made updates to the script that generates the build summary and checks for errors. One quirk is related to the current state of megaboom.

In the current run, there's a message in the build output:

[ERROR GRT-0119] Routing congestion too high. Check the congestion heatmap in the GUI and load bazel-out/k8-fastbuild/bin/reports/asap7/BoomTile/base/congestion.rpt in the DRC viewer.

But, the build continues successfully and now, this would be flagged and the build would be marked as a failure. If ORFS is reporting failure, shouldn't the ORFS run stop at that point?

For the short term, I propose to update the genReport.py script to ignore this error, so that CI doesn't report failures for this particular case.

maliberty commented 2 hours ago

ORFS will stop on error though bazel-orfs is a bit different. I think grt "passes" and then drt fails at its start.

jeffng-or commented 2 hours ago

ORFS will stop on error though bazel-orfs is a bit different. I think grt "passes" and then drt fails at its start.

In the current flow, drt isn't run. It stops with grt and the grt ODB is written out.

oharboe commented 2 hours ago

ORFS will stop on error though bazel-orfs is a bit different. I think grt "passes" and then drt fails at its start.

Indeed.

This is by intention and ref our discussion: we want bazel to generate artifacts for a failed grt, which means it has to return exit code 0, but we don't want detailed route to proceed. check out detail_route.tcl for logic.

oharboe commented 2 hours ago

ORFS will stop on error though bazel-orfs is a bit different. I think grt "passes" and then drt fails at its start.

In the current flow, drt isn't run. It stops with grt and the grt ODB is written out.

Yes, but examine the error message carefully: that error message is printed out by detail_route.tcl, no?

(If not, upgrade to latest bazel-orfs)

oharboe commented 2 hours ago

and if that doesn't help, upgrade ORFS too...