intel / llvm

Intel staging area for llvm.org contribution. Home for Intel LLVM-based projects.
Other
1.25k stars 738 forks source link

[BUG] Terrible DPC++ configuration for Linux Ubuntu machine #3823

Closed yiakwy-mapping-team closed 2 years ago

yiakwy-mapping-team commented 3 years ago

What is going on

Here is an instruction from intel how to configure latest DPC++ in a new machine:

https://intel.github.io/llvm-docs/GetStartedGuide.html

From "Build DPC++ toolchain" and "Use DPC++ toolchain" , they just don't tell it clear how to build and test the DPC++ libraries.

After compilation, I try to run one of test cases from "oneAPI-Samples" for reduce operation. It apparently failed.

Expected

  1. build it in a linux env like Ubuntu 18.04 and run it with a example. Make it as simple as few lines of automation codes in a script
  2. pass the test. Being exported and used in a CMake project

Build test failed:

OS : Ubuntu 18.04 docker OS, with necessary build essential toolkit gcc : 7.5 cmake : 3.19

[Docker-cyber_dev_base] yiakwy@yiakwy-XPS-15-9500:~/sycl_workspace$ python llvm/buildbot/check.py 
args:Namespace(base_branch=None, branch=None, build_number=None, builder_dir=None, obj_dir=None, pr_number=None, src_dir=None, test_suite='check-all')
[Cmake Command]: cmake --build /home/yiakwy/sycl_workspace/llvm/build -- check-all -j 12
[2/3] cd /home/yiakwy/sycl_workspace/llvm/clang/bindings/python && /...ycl_workspace/llvm/build/lib /usr/bin/python3.6 -m unittest discover
..............................................................................................................................
----------------------------------------------------------------------
Ran 126 tests in 0.393s

OK
[2/3] Running all regression tests
llvm-lit: /home/yiakwy/sycl_workspace/llvm/llvm/utils/lit/lit/llvm/config.py:428: note: using clang: /home/yiakwy/sycl_workspace/llvm/build/bin/clang
llvm-lit: /home/yiakwy/sycl_workspace/llvm/llvm/utils/lit/lit/llvm/config.py:428: note: using clang: /home/yiakwy/sycl_workspace/llvm/build/bin/clang
FAIL: LLVM :: tools/llvm-ranlib/D-flag.test (6208 of 72287)
******************** TEST 'LLVM :: tools/llvm-ranlib/D-flag.test' FAILED ********************
Script:
--
: 'RUN: at line 4';   /home/yiakwy/sycl_workspace/llvm/build/bin/yaml2obj /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/../llvm-ar/Inputs/add-lib1.yaml -o /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.o
: 'RUN: at line 5';   env TZ=UTC touch -t 200001020304 /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.o
: 'RUN: at line 6';   rm -f /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar cqSU /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.o
: 'RUN: at line 9';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=REAL-VALUES
: 'RUN: at line 12';   cp /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -D /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a
: 'RUN: at line 13';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=DETERMINISTIC-VALUES
: 'RUN: at line 16';   cp /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -U /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a
: 'RUN: at line 17';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=REAL-VALUES
: 'RUN: at line 20';   cp /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -UDU /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a
: 'RUN: at line 21';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=REAL-VALUES
: 'RUN: at line 22';   cp /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -UUD /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a
: 'RUN: at line 23';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=DETERMINISTIC-VALUES
: 'RUN: at line 26';   cp /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp-no-index.a /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a && /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -U /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a -D -U
: 'RUN: at line 27';   env TZ=UTC /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ar tv /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=REAL-VALUES
: 'RUN: at line 31';   not /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib --D /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a 2>&1 | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=BAD-OPT-D
: 'RUN: at line 33';   not /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib --U /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a 2>&1 | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=BAD-OPT-U
: 'RUN: at line 35';   not /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -x /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a 2>&1 | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=BAD-OPT-x
: 'RUN: at line 40';   not /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -Dx /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a 2>&1 | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=BAD-OPT-x
: 'RUN: at line 41';   not /home/yiakwy/sycl_workspace/llvm/build/bin/llvm-ranlib -DxD /home/yiakwy/sycl_workspace/llvm/build/test/tools/llvm-ranlib/Output/D-flag.test.tmp.a 2>&1 | /home/yiakwy/sycl_workspace/llvm/build/bin/FileCheck /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test --check-prefix=BAD-OPT-xD
--
Exit Code: 1

Command Output (stderr):
--
/home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test:44:25: error: DETERMINISTIC-VALUES: expected string not found in input
# DETERMINISTIC-VALUES: {{[-rwx]+}} 0/0 712 Jan 1 00:00 1970 D-flag.test.tmp.o
                        ^
<stdin>:1:1: note: scanning from here
rw-r--r-- 0/0 712 Dec 31 19:00 1969 D-flag.test.tmp.o
^

Input file: <stdin>
Check file: /home/yiakwy/sycl_workspace/llvm/llvm/test/tools/llvm-ranlib/D-flag.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
          1: rw-r--r-- 0/0 712 Dec 31 19:00 1969 D-flag.test.tmp.o 
check:44     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
>>>>>>

Suggestion

  1. Most libraries in Linux is using gcc not clang.
  2. Few people now are developing libraries in Make, please switch to CMake or Bazel system.
  3. Tell instructions with complete examples. A dockerfile is better.

Also see https://github.com/oneapi-src/oneDPL/issues/256#issuecomment-848595269

yiakwy-mapping-team commented 3 years ago

@romanovvlad

AlexeySachkov commented 3 years ago

Hi @yiakwy-mapping-team, thanks for you bug report

A few questions:

After compilation, I try to run one of test cases from "oneAPI-Samples" for reduce operation. It apparently failed.

Could you please attach corresponding logs and reproducer? There is an example of the failed test, but it has nothing to do with DPC++ - it is just a test for llvm-ranlib utility which is not used by the toolchain as far as I know.

build it in a linux env like Ubuntu 18.04 and run it with a example. Make it as simple as few lines of automation codes in a script

From my point of view this is exactly what we have right now: according tp Build DPC++ toolchain section you need to launch configure.py script and then compile.py script and that would be all.

Most libraries in Linux is using gcc not clang.

What is the exact suggestion here? You should be able to compile DPC++ toolchain by both gcc and clang

Few people now are developing libraries in Make, please switch to CMake or Bazel system.

We do not use make directly, we use CMake and you can choose the particular build system by selecting corresponding CMake generator through --cmake-gen option of configure.py

Tell instructions with complete examples.

This is also present in Get Started Guide, see Run simple DPC++ application

yiakwy-mapping-team commented 3 years ago

@alexey-bataev

  1. I will give you full message print onto the screen soon. To reproduce the screen, just follow the steps (misunderstanding might happen here) and check the tests
  2. It became confusing when I went to section

    Use DPC++ toolchain

  3. Yes I can compile but it is not very clear what which version of gcc used by the project.
  4. I mean there is no FindDPC shipped. How I can use them in a third-party cmake system?
MrSidims commented 3 years ago

Yes I can compile but it is not very clear what which version of gcc used by the project.

According to https://intel.github.io/llvm-docs/GetStartedGuide.html#prerequisites

Linux: GCC version 7.1.0 or later (including libstdc++)

Or are we talking about device/host compilation of the actual DPCPP code and not compilation of the compilator? I'm confused.

alexbatashev commented 3 years ago

I mean there is no FindDPC shipped. How I can use them in a third-party cmake system?

Just like you'd use any other compiler: either by setting CC=clang and CXX=clang++ (on Linux), or by passing -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ (also linux) or -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl (windows) flags to cmake configuration line. Just make sure that clang is actually clang executable, that was result of building this repository code.

AlexeySachkov commented 3 years ago

I mean there is no FindDPC shipped. How I can use them in a third-party cmake system?

Yes, this is a missing thing for sure. Related feature request is #1000 - if you have particular suggestions feel free to post them there.

I see that there was an attempt to add FindSYCL into CMake distribution here, but it doesn't seem to be finished due to some design concerns.

Meanwhile, you can configure C/CXX compiler as @alexbatashev suggested. Here is an example of how it is done in official oneAPI samples.

AlexeySachkov commented 2 years ago

@yiakwy-mapping-team, is there something else you would like to ask/clarify? Can we close this issue?

AlexeySachkov commented 2 years ago

I assume that we have answered all questions and therefor I'm closing the issue. @yiakwy-mapping-team, if that is not the case, feel free to re-open or open a new one for further questions