search

efabless / globalfoundries-pdk-libs-gf180mcu_fd_pr

Primitives for GF180MCU provided by GlobalFoundries.
https://gf180mcu-pdk.rtfd.io
Apache License 2.0
10 stars 8 forks source link

deep mode error #13

Closed marwaneltoukhy closed 1 year ago

marwaneltoukhy commented 1 year ago

Expected Behavior

No errors

Actual Behavior

Prints an nonfatal error:

ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true                                                                                                   
ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0                                                                                                                                               
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true                                                                                                   
ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0                                                                                                                                               
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true                                                                                                   
ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0                                                                                                                                               
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true                                                                                                   
ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0                                                                                                                                               
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true                                                                                                   
ERROR: ../../../src/db/db/dbRegionLocalOperations.cc,839,nstart > 0                                                                                                                                               
ERROR: Worker thread: Internal error: ../../../src/db/db/dbRegionLocalOperations.cc:839 nstart > 0 was not true

Steps to Reproduce the Problem

  1. python3 run_drc.py --thr=1 --variant=C --path=/home/marwan/caravel-gf180mcu/macros/simple_por/gds/simple_por.gds --run_dir=/home/marwan/caravel-gf180mcu/signoff/simple_por/standalone_pvr --run_mode=deep

Specifications

atorkmabrains commented 1 year ago

@klayoutmatthias Could you please check that? I believe the deep mode is not working properly.

@marwaneltoukhy Please don't use deep mode in your runs.

klayoutmatthias commented 1 year ago

I never got this error myself, but I'll check.

Where can I find the test case for "/home/marwan/caravel-gf180mcu/macros/simple_por/gds/simple_por.gds"? In the DRC deck is from the master branch?

And 0.28 is kind of unspecific as it is under development. If possible, indicate the revision you used.

Thanks,

Matthias

klayoutmatthias commented 1 year ago

I think I found the test case.

But wasn't able to reproduce the problem.

The DRC finishes with no errors after about 8s.

Matthias

klayoutmatthias commented 1 year ago

@marwaneltoukhy Wild guess: are you using KLayout wip branch? Please don't. It is work in progress. The master branch is the best choice for you and already contains the major performance improvements.

atorkmabrains commented 1 year ago

@klayoutmatthias I'm using master klayout and getting the same issue. here:

python3 run_drc.py --variant=C --path=../../../../../caravel-gf180mcu/macros/simple_por/gds/simple_por.gds.gz --run_dir=./drc_runs --run_mode=deep

Here is my log

04-Dec-2022 20:32:24 | INFO    | Your Klayout version is: KLayout 0.28
04-Dec-2022 20:32:24 | INFO    | ## Generating template with for the following rule tables: ['otp_mk.drc', 'poly2.drc', 'esd.drc', 'lres.drc', 'via1.drc', 'metaltop.drc', 'lvs_bjt.drc', 'mim.drc', 'pplus.drc', 'dnwell.drc', 'nplus.drc', 'drc_bjt.drc', 'via4.drc', 'ldpmos.drc', 'ymtp_mk.drc', 'via5.drc', 'via2.drc', 'efuse.drc', 'geom.drc', 'contact.drc', 'pres.drc', 'metal5.drc', 'dualgate.drc', 'sram.drc', 'sab.drc', 'metal1.drc', 'metal2.drc', 'metal4.drc', 'via3.drc', 'hres.drc', 'lvpwell.drc', 'mcell.drc', 'comp.drc', 'metal3.drc', 'nat.drc', 'dummy_exclude.drc', 'ldnmos.drc', 'nwell.drc']
../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/drc_runs
04-Dec-2022 20:32:24 | INFO    | Running Global Foundries 180nm MCU ../gf180mcu_project/caravel-gf180mcu/macros/simple_por/gds/simple_por.gds.gz checks on design main on cell simple_por:
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Starting running GF180MCU Klayout DRC runset on ../gf180mcu_project/caravel-gf180mcu/macros/simple_por/gds/simple_por.gds.gz
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Ruby Version for klayout: 3.0.2
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Loading database to memory is complete.
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : GF180MCU Klayout DRC runset output at:../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/drc_runs/simple_por_main.lyrdb
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Number of threads to use 16
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Verbose mode: false
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : deep  mode is enabled.
2022-12-04 20:32:25 +0200: Memory Usage (582644K) : Read in polygons from layers.
2022-12-04 20:32:25 +0200: Memory Usage (588596K) : Starting deriving base layers.
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : Evaluate switches.
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : FEOL is enabled.
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : BEOL is enabled.
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : connectivity rules are disabled.
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : METAL_TOP Selected is 9K
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : METAL_STACK Selected is 5LM
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : Wedge enabled  true
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : Ball enabled  true
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : Gold enabled  true
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : MIM Option selected B
2022-12-04 20:32:25 +0200: Memory Usage (2719244K) : Offgrid enabled  true
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Total area of the design is 5481.4924 um^2.
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Total no. of polygons in the design is 5686
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Initialization and base layers definition.
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Starting GF180MCU DRC rules.
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Running all FEOL rules
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Running all BEOL rules
2022-12-04 20:32:25 +0200: Memory Usage (2719380K) : Executing rule O.DF.3a
2022-12-04 20:32:25 +0200: Memory Usage (2719528K) : Executing rule O.DF.6
2022-12-04 20:32:25 +0200: Memory Usage (2719804K) : Executing rule O.DF.9
2022-12-04 20:32:25 +0200: Memory Usage (2719804K) : Executing rule O.PL.2
addr2line: 'klayout': No such file
ERROR: Signal number: 11
Address: 0x0
Program Version: KLayout 0.28 (2022-12-02 rb3968c31e)

Backtrace:
/tools/klayout-v0.28_dev/bin-release/libklayout_lay.so.0 +0x2dc357 lay::enable_signal_handler_gui(bool) [??:?]
/lib/x86_64-linux-gnu/libc.so.6 +0x42520 __restore_rt [libc_sigaction.c:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0x60f358 db::Cell::begin_parent_cells() const [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0x13680d9 db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >::compute_results(db::local_processor_contexts<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >&, db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> > const*, std::vector<unsigned int, std::allocator<unsigned int> > const&) const [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0x13688c1 db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >::run(db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >*, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0x1368b41 db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >::run(db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::edge<int> >*, unsigned int, unsigned int, unsigned int) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0x137dba9 db::DeepRegion::edges(db::EdgeFilterBase const*) const [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0xf604f8 db::Region::edges() const [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_db.so.0 +0xfbbf31 gsi::VectorAdaptorImpl<std::vector<db::Region, std::allocator<db::Region> > >::push(gsi::SerialArgs&, tl::Heap&) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_rba.so.0 +0x2974f rba::RubyStackTraceProvider::stack_trace() const [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x22fc45 rb_dtrace_setup [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23ccb1 rb_vm_invoke_bmethod [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x231fd6 rb_vm_localjump_error [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x234c42 rb_check_funcall [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23a31d rb_vm_exec [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23acd7 rb_vm_exec [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23e128 rb_mod_module_exec [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x22fc45 rb_dtrace_setup [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x231fd6 rb_vm_localjump_error [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x234c42 rb_check_funcall [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23a31d rb_vm_exec [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x24294c rb_funcallv [??:?]
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0xa69bd rb_protect [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_rba.so.0 +0x48915 void std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> > >::_M_realloc_insert<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> >(__gnu_cxx::__normal_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> > > >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>&&) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_rba.so.0 +0x4c177 tl::BacktraceElement::~BacktraceElement() [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_gsi.so.0 +0xa73b4 void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_gsi.so.0 +0xa769e void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_tl.so.0 +0xc7f00 tl::Executable::do_execute() [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_lym.so.0 +0x4b874 lym::MacroInterpreter::execute_macro(lym::Macro const*) [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_lym.so.0 +0x562a5 lym::Macro::run() const [??:?]
/tools/klayout-v0.28_dev/bin-release/libklayout_lay.so.0 +0x1550b7 lay::ApplicationBase::run() [??:?]
klayout +0x4d95
/tools/klayout-v0.28_dev/bin-release/libklayout_rba.so.0 +0x24ac4
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x22fc45
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x231fd6
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x234c42
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0x23a31d rb_vm_exec
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0xa5328
/lib/x86_64-linux-gnu/libruby-3.0.so.3.0 +0xa8e2a ruby_run_node
/tools/klayout-v0.28_dev/bin-release/libklayout_rba.so.0 +0x26242 _ZN3rba15RubyInterpreter10initializeERiPPcPFiS1_S3_E
klayout +0x489b
klayout +0x3fb7
/lib/x86_64-linux-gnu/libc.so.6 +0x29d90
/lib/x86_64-linux-gnu/libc.so.6 +0x29e40 __libc_start_main
klayout +0x4075
Traceback (most recent call last):
  File "../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/run_drc.py", line 534, in <module>
    main(drc_run_dir, now_str, arguments)
  File "../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/run_drc.py", line 498, in main
    list_res_db_files.append(run_check(drc_file, "main", layout_path, drc_run_dir, switches))
  File "../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/run_drc.py", line 424, in run_check
    check_call(run_str, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'klayout -b -r ../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/drc_runs/main.drc -rd thr=16 -rd run_mode=deep -rd metal_top=9K -rd mim_option=B -rd metal_level=5LM -rd feol=true -rd beol=true -rd offgrid=true -rd conn_drc=false -rd density=false -rd topcell=simple_por -rd input=../gf180mcu_project/caravel-gf180mcu/macros/simple_por/gds/simple_por.gds.gz -rd report=../gf180mcu_project/mabrains_version/globalfoundries-pdk-libs-gf180mcu_fd_pr/rules/klayout/drc/drc_runs/simple_por_main.lyrdb ' returned non-zero exit status 11.
klayoutmatthias commented 1 year ago

This is a segfault, not the internal error above.

Looks like the default is 16 threads ... the stack trace gives some faint hints what may be wrong, but this is very difficult to debug.

What happens with "--thr=1"?

Matthias

klayoutmatthias commented 1 year ago

@atorkmabrains Cool, I can reproduce the problem, but only with 16 threads. Looks like a race condition.

For me it works with 1 and 4 threads.

Matthias

marwaneltoukhy commented 1 year ago

@marwaneltoukhy Wild guess: are you using KLayout wip branch? Please don't. It is work in progress. The master branch is the best choice for you and already contains the major performance improvements.

That is correct, tried using master and it worked with 4 threads. Thank you!

NOTE: will keep this issue open to track the deep mode issue for more than 4 threads

klayoutmatthias commented 1 year ago

I guess I found the problem for the thread issue - a lock was missing. I'll update master with the threading patch. --thr=1 is always safe.

But so far, I cannot reproduce the internal error reported above. That is a different thing.

I'll keep trying.

Matthias

klayoutmatthias commented 1 year ago

@marwaneltoukhy I just tried to reproduce the internal error you found initially on the wip branch without success.

I used Debian 11.5 in a Docker instance with klayout wip branch (commit 43ff59750ab17fe3c3a13f504c03a7a1244a6f68). Layout from https://raw.githubusercontent.com/efabless/caravel-gf180mcu/main/macros/simple_por/gds/simple_por.gds.gz.

Call was: 

python3 run_drc.py --variant=C --path=/simple_por.gds.gz --run_dir=/simple_por_drc --run_mode=deep --thr=1

No error, just clean DRC :(

I am quite interested to debug this problem as a want to merge the wip branch sooner or later. Can you confirm I got the right GDS file?

Thanks and best regards,

Matthias

klayoutmatthias commented 1 year ago

0.28 has been released - although I could not reproduce the initial problem.

Maybe you like to try that or close this issue.

Matthias

atorkmabrains commented 1 year ago

This is now fixed with version 0.28 release.