Illegal instruction hit when using conda-packages with netgen-enabled libmesh

[GiudGiud]

Bug Description

Some users are hitting this when updating moose-libmesh to our late July and mid-August versions:

(moose) giudgl@FN651821:~/projects/moose/modules/reactor/test/tests/meshgenerators/peripheral_ring_mesh_generator$ ../../../../reactor-devel -i ring_quadratic.i --mesh-only
Illegal instruction: 4

I got a backtrace on intel-mac. Others have hit this on M1 and on linux

(moose) giudgl@FN651821:~/projects/moose/modules/reactor/test/tests/meshgenerators/peripheral_ring_mesh_generator$ lldb -- ../../../../reactor-devel -i ring_quadratic.i --mesh-only
(lldb) target create "../../../../reactor-devel"
Current executable set to '/Users/giudgl/projects/moose/modules/reactor/reactor-devel' (x86_64).
(lldb) settings set -- target.run-args  "-i" "ring_quadratic.i" "--mesh-only"
(lldb) run
Process 38739 launched: '/Users/giudgl/projects/moose/modules/reactor/reactor-devel' (x86_64)
Process 38739 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00000001010b5050 libngcore.dylib`_GLOBAL__sub_I_profiler.cpp + 368
libngcore.dylib`:
->  0x1010b5050 <+368>: kxnorw %k0, %k0, %k1
    0x1010b5054 <+372>: vscatterdpd %zmm0, (%rcx,%ymm1) {%k1}
    0x1010b505b <+379>: kxnorw %k0, %k0, %k1
    0x1010b505f <+383>: vpscatterdd %ymm2, 0x48(%rcx,%ymm1) {%k1}
Target 0: (reactor-devel) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
  * frame #0: 0x00000001010b5050 libngcore.dylib`_GLOBAL__sub_I_profiler.cpp + 368
    frame #1: 0x00007ff807e2739f dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 185
    frame #2: 0x00007ff807e614c8 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 133
    frame #3: 0x00007ff807e55bcf dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 543
    frame #4: 0x00007ff807e0c07f dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 249
    frame #5: 0x00007ff807e54d0c dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 176
    frame #6: 0x00007ff807e571b0 dyld`dyld3::MachOFile::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, bool&) block_pointer) const + 116
    frame #7: 0x00007ff807e611fe dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 390
    frame #8: 0x00007ff807e240cd dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 225
    frame #9: 0x00007ff807e299c9 dyld`dyld4::JustInTimeLoader::runInitializers(dyld4::RuntimeState&) const + 21
    frame #10: 0x00007ff807e242e5 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 213
    frame #11: 0x00007ff807e2428f dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 127
    frame #12: 0x00007ff807e2428f dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 127
    frame #13: 0x00007ff807e2428f dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 127
    frame #14: 0x00007ff807e2428f dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 127
    frame #15: 0x00007ff807e2428f dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 127
    frame #16: 0x00007ff807e27414 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 98
    frame #17: 0x00007ff807e2437e dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 96
    frame #18: 0x00007ff807e43d9a dyld`dyld4::APIs::runAllInitializersForMain() + 326
    frame #19: 0x00007ff807e11203 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3364
    frame #20: 0x00007ff807e102ff dyld`start + 1839

Steps to Reproduce

Build libmesh on civet with netgen Download from the PRs ' dedicated conda channel Use those packages on a problematic machine (this part is random)

On my machine with the issue:

• build from source w/ netgen worked
• build from locally built conda packages w/ netgen worked

Impact

Unable to update moose Have to revert to an old version from mid July

[Optional] Diagnostics

No response

[lindsayad]

was this fixed with the libmesh update?

[GiudGiud]

will verify this as soon as there s a next merge and a package release

[GiudGiud]

I confirm this is fixed on my intel machine Closing as resolved by #28508