llvmbot
commented
1 year ago @llvm/issue-subscribers-openmp
Closed lixi-zhou closed 1 year ago
llvmbot
commented
1 year ago @llvm/issue-subscribers-openmp
llvmbot
commented
1 year ago @llvm/issue-subscribers-backend-x86
shiltian
commented
1 year ago LLVM 14 is fairly out of date now. I fixed some failures on macOS in the latest release. Can you try that? We will not fix any LLVM 14 failure.
kkwli
commented
1 year ago I get the same failures when building from 4638eb26603c8d1cd3c46167734356e2585afdfe.
lixi-zhou
commented
1 year ago LLVM 14 is fairly out of date now. I fixed some failures on macOS in the latest release. Can you try that? We will not fix any LLVM 14 failure.
Still get the same test failures with the build from the latest release 16.0.5.
Should the backend be marked as arm64 instead of x86? @EugeneZelenko
llvmbot
commented
1 year ago @llvm/issue-subscribers-backend-aarch64
shiltian
commented
1 year ago I'm gonna land a patch (https://reviews.llvm.org/D159324). On my Intel-based macOS, I can only see one failure: ompt/misc/interoperability.cpp. Can you give it a shot and let me know if barrier/omp_barrier.c and parallel/omp_parallel_if.c still fail? If so, I'm afraid I can't fix it because I don't have access to M series macOS.
kkwli
commented
1 year ago With 35fdf8d built on M1 Max, I still see the following failures:
libomp :: barrier/omp_barrier.c
libomp :: ompt/synchronization/masked.c
libomp :: ompt/synchronization/master.c
libomp :: parallel/omp_parallel_if.cWith
35fdf8dbuilt on M1 Max, I still see the following failures:libomp :: barrier/omp_barrier.c libomp :: ompt/synchronization/masked.c libomp :: ompt/synchronization/master.c libomp :: parallel/omp_parallel_if.c
Thanks for the info, then unfortunately I'm unable to work on it.
shiltian
commented
1 year ago Okay, the issue is caused by https://reviews.llvm.org/D138495#inline-1543447. Long story short, it looks like there is some ABI difference between X86 and ARM64 such that the last (3rd) argument is not stored in the corresponding register x2, while in the callee it tries to read the last argument. I'm still looking into it.
0x10021a84c <+188>: 0x910053e0 add x0, sp, #0x14
0x10021a850 <+192>: 0x910043e1 add x1, sp, #0x10
0x10021a854 <+196>: 0xd63f0100 blr x8
-> 0x10021a858 <+200>: 0x14000006 b 0x10021a870 ; <+224> at kmp_csupport.cpp:360:36Meanwhile in the callee:
0x100003d18 <+0>: 0xd10103ff sub sp, sp, #0x40
0x100003d1c <+4>: 0xa9037bfd stp x29, x30, [sp, #0x30]
0x100003d20 <+8>: 0x9100c3fd add x29, sp, #0x30
-> 0x100003d24 <+12>: 0xf9400048 ldr x8, [x2]parallel/omp_parallel_if.c is passing now.
barrier/omp_barrier.c is still failing.
shiltian
commented
1 year ago Hmm, interesting. I have all of them pass on my M2 Ultra.
kkwli
commented
1 year ago Both parallel/omp_parallel_if.c and barrier/omp_barrier.c are passing now. 🎊🎉
Thanks a lot @shiltian.
lixi-zhou
commented
1 year ago Hi, @shiltian, many thanks for your efforts in fixing this. With 010a5a7 built on M1 Pro chip, the libomp :: barrier/omp_barrier.c and libomp :: parallel/omp_parallel_if.c get passed. While I still got the following two tests failed. Did you get these two passed at your side?
libomp :: ompt/synchronization/masked.c
libomp :: ompt/synchronization/master.cDetailed results are attached below
[0/1] Running OpenMP tests
-- Testing: 318 tests, 8 workers --
Testing:
FAIL: libomp :: ompt/synchronization/master.c (1 of 318)
******************** TEST 'libomp :: ompt/synchronization/master.c' FAILED ********************
Exit Code: 1
Command Output (stdout):
--
# RUN: at line 1
/Users/macbook/GitHub/llvm-project/build/./bin/clang -fopenmp -I /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/src -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test -L /Users/macbook/GitHub/llvm-project/build/./lib -fno-omit-frame-pointer -Wl,-rpath,/Users/macbook/GitHub/llvm-project/build/./lib -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c -o /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp -lm && /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp | tee /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp.out | /Users/macbook/GitHub/llvm-project/build/./bin/FileCheck /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c
# executed command: /Users/macbook/GitHub/llvm-project/build/./bin/clang -fopenmp -I /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/src -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test -L /Users/macbook/GitHub/llvm-project/build/./lib -fno-omit-frame-pointer -Wl,-rpath,/Users/macbook/GitHub/llvm-project/build/./lib -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c -o /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp -lm
# executed command: /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp
# executed command: tee /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/master.c.tmp.out
# executed command: /Users/macbook/GitHub/llvm-project/build/./bin/FileCheck /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c
# .---command stderr------------
# | /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c:89:11: error: CHECK: expected string not found in input
# | // CHECK: {{^}}[[MASTER_ID]]: current_address={{.*}}[[RETURN_ADDRESS_END]]
# | ^
# | <stdin>:4:49: note: scanning from here
# | 0: ompt_event_master_end: codeptr_ra=0x10271bb6c
# | ^
# | <stdin>:4:49: note: with "MASTER_ID" equal to "0"
# | 0: ompt_event_master_end: codeptr_ra=0x10271bb6c
# | ^
# | <stdin>:4:49: note: with "RETURN_ADDRESS_END" equal to "0x10271bb6c"
# | 0: ompt_event_master_end: codeptr_ra=0x10271bb6c
# | ^
# | <stdin>:5:1: note: possible intended match here
# | 0: current_address=0x10271bb74 or 0x10271bb70
# | ^
# |
# | Input file: <stdin>
# | Check file: /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/master.c
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# | 1: 0: NULL_POINTER=0x0
# | 2: 0: ompt_event_master_begin: codeptr_ra=0x10271bad4
# | 3: 0: fuzzy_address=0x10271b9 or 0x10271ba or 0x10271bb or 0x10271bc (0x10271baec)
# | 4: 0: ompt_event_master_end: codeptr_ra=0x10271bb6c
# | check:89'0 X error: no match found
# | check:89'1 with "MASTER_ID" equal to "0"
# | check:89'2 with "RETURN_ADDRESS_END" equal to "0x10271bb6c"
# | 5: 0: current_address=0x10271bb74 or 0x10271bb70
# | check:89'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:89'3 ? possible intended match
# | 6: 0: current_address=0x10271bb74 or 0x10271bb70
# | check:89'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | 7: 0: x=1
# | check:89'0 ~~~~~~~
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1
--
********************
Testing:
FAIL: libomp :: ompt/synchronization/masked.c (2 of 318)
******************** TEST 'libomp :: ompt/synchronization/masked.c' FAILED ********************
Exit Code: 1
Command Output (stdout):
--
# RUN: at line 1
/Users/macbook/GitHub/llvm-project/build/./bin/clang -fopenmp -I /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/src -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test -L /Users/macbook/GitHub/llvm-project/build/./lib -fno-omit-frame-pointer -Wl,-rpath,/Users/macbook/GitHub/llvm-project/build/./lib -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c -o /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp -lm && /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp | tee /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp.out | /Users/macbook/GitHub/llvm-project/build/./bin/FileCheck /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c
# executed command: /Users/macbook/GitHub/llvm-project/build/./bin/clang -fopenmp -I /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/src -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test -L /Users/macbook/GitHub/llvm-project/build/./lib -fno-omit-frame-pointer -Wl,-rpath,/Users/macbook/GitHub/llvm-project/build/./lib -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c -o /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp -lm
# executed command: /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp
# executed command: tee /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp.out
# executed command: /Users/macbook/GitHub/llvm-project/build/./bin/FileCheck /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c
# .---command stderr------------
# | /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c:38:11: error: CHECK: expected string not found in input
# | // CHECK: {{^}}[[MASTER_ID]]: current_address={{.*}}[[RETURN_ADDRESS_END]]
# | ^
# | <stdin>:8:117: note: scanning from here
# | 281474976710658: ompt_event_masked_end: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d66c
# | ^
# | <stdin>:8:117: note: with "MASTER_ID" equal to "281474976710658"
# | 281474976710658: ompt_event_masked_end: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d66c
# | ^
# | <stdin>:8:117: note: with "RETURN_ADDRESS_END" equal to "0x10040d66c"
# | 281474976710658: ompt_event_masked_end: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d66c
# | ^
# | <stdin>:9:13: note: possible intended match here
# | 281474976710658: current_address=0x10040d674 or 0x10040d670
# | ^
# |
# | Input file: <stdin>
# | Check file: /Users/macbook/GitHub/llvm-project/openmp/runtime/test/ompt/synchronization/masked.c
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# | 1: 0: NULL_POINTER=0x0
# | 2: 281474976710658: ompt_event_thread_begin: thread_type=ompt_thread_initial=1, thread_id=281474976710658
# | 3: 281474976710658: ompt_event_initial_task_begin: parallel_id=281474976710660, task_id=281474976710659, actual_parallelism=1, index=1, flags=1
# | 4: 281474976710658: ompt_event_parallel_begin: parent_task_id=281474976710659, parent_task_frame.exit=0x0, parent_task_frame.reenter=0x16f9f7490, parallel_id=281474976710661, requested_team_size=2, codeptr_ra=0x10040d540, invoker=2
# | 5: 281474976710658: ompt_event_implicit_task_begin: parallel_id=281474976710661, task_id=281474976710662, team_size=2, thread_num=0
# | 6: 281474976710658: ompt_event_masked_begin: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d5d4
# | 7: 281474976710658: fuzzy_address=0x10040d4 or 0x10040d5 or 0x10040d6 or 0x10040d7 (0x10040d5ec)
# | 8: 281474976710658: ompt_event_masked_end: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d66c
# | check:38'0 X error: no match found
# | check:38'1 with "MASTER_ID" equal to "281474976710658"
# | check:38'2 with "RETURN_ADDRESS_END" equal to "0x10040d66c"
# | 9: 281474976710658: current_address=0x10040d674 or 0x10040d670
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:38'3 ? possible intended match
# | 10: 281474976710658: ompt_event_barrier_begin: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d540
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | 11: 281474976710658: task level 0: parallel_id=281474976710661, task_id=281474976710662, exit_frame=0x0, reenter_frame=0x0, task_type=ompt_task_implicit|ompt_task_undeferred=134217730, thread_num=0
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | 12: 281474976710658: ompt_event_wait_barrier_begin: parallel_id=281474976710661, task_id=281474976710662, codeptr_ra=0x10040d540
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | 13: 562949953421313: ompt_event_thread_begin: thread_type=ompt_thread_worker=2, thread_id=562949953421313
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | 14: 562949953421313: ompt_event_implicit_task_begin: parallel_id=281474976710661, task_id=562949953421314, team_size=2, thread_num=1
# | check:38'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | .
# | .
# | .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1
--
********************
Testing: 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (2):
libomp :: ompt/synchronization/masked.c
libomp :: ompt/synchronization/master.c
Testing Time: 158.68s
Unsupported : 24
Passed : 291
Expectedly Failed: 1
Failed : 2
FAILED: projects/openmp/CMakeFiles/check-openmp /Users/macbook/GitHub/llvm-project/build/projects/openmp/CMakeFiles/check-openmp
cd /Users/macbook/GitHub/llvm-project/build/projects/openmp && /Users/macbook/miniconda3/bin/python3.8 /Users/macbook/GitHub/llvm-project/build/./bin/llvm-lit -sv /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test
ninja: build stopped: subcommand failed.
@lixi-zhou Yeah, the two OMPT tests also fail on my system, but I have no idea of how OMPT works. @jprotze is our OMPT expert.
jprotze
commented
1 year ago @lixi-zhou given the directory names, I assume you run the tests locally? The following should give us the necessary insights for the generated code:
objdump -d /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp | grep -A10 '__kmpc_end_master'If you don't have objdump, try llvm-objdump from your llvm installation.
My suspicion is that on your system clang inserts an additional instruction between __kmpc_end_master and the nop.
printf("%" PRIu64 ": current_address=%p or %p or %p\n", ompt_get_thread_data()->value, \
((char *)addr) - 4, ((char *)addr) - 8, ((char *)addr) - 12)I don't care, whether you limit this change to mac os or whether you print this additional address for all AArch64 systems.
lixi-zhou
commented
1 year ago This command output nothing from my side
objdump -d /Users/macbook/GitHub/llvm-project/build/projects/openmp/runtime/test/ompt/synchronization/Output/masked.c.tmp | grep -A10 '__kmpc_end_master'I attached the disassembled masked.c.tmp file below for your reference.
jprotze
commented
1 year ago I have no experience with the AA64 instruction set. I assume that bl effectively implements the function call. From the test output you posted above, the test tries to match 0x10271bb6c with current_address=0x10271bb74 or 0x10271bb70. That's why I suggested to additionally print ((char *)addr) - 12 which would result in current_address=0x10271bb74 or 0x10271bb70 or 0x10271bb6c.
We insert a label and a nop instruction. Looking at your disassembled code, I would expect a difference of 16, but as I said, I have no experience with AA64. The relevant instructions should be:
1000055d0: 94000044 bl 0x1000056e0 <_usleep+0x1000056e0>
1000055d4: 71000008 subs w8, w0, #0x0
1000055d8: 1a9f17e8 cset w8, eq
1000055dc: 370004a8 tbnz w8, #0x0, 0x100005670 <_main.omp_outlined+0xf0>
1000055e0: 14000001 b 0x1000055e4 <_main.omp_outlined+0x64>
1000055e4: d503201f nopCould you test the change I suggested?
lixi-zhou
commented
1 year ago Hi, @jprotze, thank you for your prompt reply. After changing this, I can confirm that both two tests are passed.
jprotze
commented
1 year ago Feel free to push this change as NFC commit. Otherwise I'll do so next week.
lixi-zhou
commented
1 year ago Thanks. I created one PR for the fix. #65696
Feel free to push this change as NFC commit. Otherwise I'll do so next week.
Run test cases of OpenMP on M1 Mac with clang 14.0.3, getting the following failed test cases
Steps to reproduce:
Detailed console output: