How to build applications with ARM SVE support using buildroot toolchains?

guoqinglei commented 5 years ago

For example, for HPC applications, I want to turn on the ARM SVE compiler option, How Can I achieve this ?

Take the dhrystone for example, I use the following command:

./build-buildroot --config 'BR2_PACKAGE_DHRYSTONE=y' --config 'BR2_OPTIMIZE_3=y'

How can I make sure that the sve compiler option has been opened and How can I know that the generated executable has the sve instructions ?

cirosantilli commented 5 years ago

EDIT: I have added a concrete SVE example at: https://cirosantilli.com/linux-kernel-module-cheat/#update-gcc-gcc-supported-by-buildroot at LKMC 6936bd6ba996dee40f7cd826e5cf01ef39c2cabf

According to GCC release notes https://www.gnu.org/software/gcc/gcc-8/changes.html GCC 8 has support and you have to pass:

gcc -march=armv8.2-a+sve

We can check our GCC version with:

./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-gcc --version

and at master 31473ebb398f9a2b638b04c14b456ffa312ac667 it gives:

7.3.0

so there is no support on LKMC master.

Buildroot 2019.08 does have GCC 8 by default however, so one easy thing to do is to update to that buildroot and rebuild everything. I want to do this at some point when I get some time, it would also solve other known bugs.

Alternatively, if lazy, Ubuntu 19.04 already has aarch64-linux-gnu-gcc, you could try just compiling it statically with that GCC and see if it works. Not guaranteed but likely.

I don't think there is an easier way to see if SVE instructions are used than just looking for SVE instructions on objdump disassembly or comparing an SVE vs non SVE build to find those instructions.

This should be easy, have a look at this SVE assembly example: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/31473ebb398f9a2b638b04c14b456ffa312ac667/userland/arch/aarch64/sve.S#L47 and notice that most SVE instructions have z1.d, z2.d, etc. or p0/z, p0/m.

guoqinglei commented 5 years ago

Actually, My Buildroot version is 2018.08, and the gcc version is still 7.3.0, as follows:

tom@tom-pc:~/git/linux-kernel-module-cheat$ ./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-gcc --version
aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2018.08-g653eaa17) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I wonder if I should update the gcc to 8.0 individually in submodule/gcc and rebuild the buildroot ?

I see the git log of submodule/gcc:

tom@tom-pc:~/git/linux-kernel-module-cheat/submodules/gcc$ git log
commit 9046071dddb0a9ee7e30e5edf27cfc53b5b9c238 (grafted, HEAD)
Author: ktkachov <ktkachov@138bc75d-0d04-0410-961f-82ee72b054a4>
Date:   Tue Jun 5 09:50:16 2018 +0000

    PR target/81497: Fix arm_acle.h for C++

    When trying to compile something with arm_acle.h using G++ we get a number of nasty errors:
    arm_acle.h:48:49: error: invalid conversion from ‘const void*’ to ‘const int*’ [-fpermissive]
        return __builtin_arm_ldc (__coproc, __CRd, __p);

    This is because the intrinsics that are supposed to be void return the "result" of their builtin,
    which is void. C lets that slide but C++ complains.

    After fixing that we run into further errors:
    arm_acle.h:48:46: error: invalid conversion from 'const void*' to 'const int*' [-fpermissive]
        return __builtin_arm_ldc (__coproc, __CRd, __p);
                                                   ^~~
    Because the pointer arguments in these intrinsics are void pointers but the builtin
    expects int pointers. So this patch introduces new qualifiers for void pointers and their
    const-qualified versions and uses that in the specification of these intrinsics.

    This gives us the opportunity of creating an arm subdirectory in g++.dg and inaugurates it
    with the first arm-specific C++ tests (in that directory).

            PR target/81497
            * config/arm/arm-builtins.c (arm_type_qualifiers): Add
            qualifier_void_pointer and qualifier_const_void_pointer.
            (arm_ldc_qualifiers, arm_stc_qualifiers): Use the above.
            (arm_init_builtins): Handle the above.
            * config/arm/arm_acle.h (__arm_cdp, __arm_ldc, __arm_ldcl, __arm_stc,
            __arm_stcl, __arm_mcr, __arm_cdp2, __arm_ldc2, __arm_ldcl2, __arm_stc2,
            __arm_stcl2,__arm_mcr2, __arm_mcrr, __arm_mcrr2): Remove return for
            void intrinsics.

            * g++.target/arm/arm.exp: New file.
            * g++.target/arm/pr81497.C: Likewise.

    git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@261191 138bc75d-0d04-0410-961f-82ee72b054a4
    Upstream-Status: Merged (gcc-8-branch)
    Signed-off-by: Gaël PORTAY <gael.portay@savoirfairelinux.com>
    [gportay: drop gcc/{,testsuite/}ChangeLog changes]

According to the git log, the gcc is already gcc 8.0 ?

cirosantilli commented 5 years ago

Where in that log does it say that it is 8.0? :-)

If I go back a few commits I see the tag 87fb575328cc5d954b91672681aacfc383134b12 (tag: gcc-7_3_0-release).

Things would likely work if you updated GCC and forced it to rebuild. But then be careful because if another build is made, likely some package could fail to build due to different GCC version.

This would be better if we had a --gcc-build-id or --<submodule>-build-id flag for all submodules, but we don't currently.

guoqinglei commented 5 years ago

I have tried to rebuild buildroot with gcc 8.* verison. And run the command: ./build-buildroot --config-fragment ./buildroot_config/config_aarch64_gcc8.2 --force-rebuild

the file ./buildroot_config/config_aarch64_gcc8.2 include some configs of gcc version set. I have also install gcc 8. in ubuntu 18.04 LTS, and the default gcc is gcc 8.. The error is as follows:

>>> host-attr 2.4.48 Patching

Applying 0001-build-with-older-GCCs.patch using patch: 
Error: duplicate filename '0001-build-with-older-GCCs.patch'
Conflicting files are:
  already applied: /home/tom/git/linux-kernel-module-cheat/submodules/buildroot/package/attr/0001-build-with-older-GCCs.patch
  to be applied  : /home/tom/git/linux-kernel-module-cheat/submodules/buildroot/package/attr/0001-build-with-older-GCCs.patch
package/pkg-generic.mk:193: recipe for target '/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/default/aarch64/build/host-attr-2.4.48/.stamp_patched' failed
make: *** [/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/default/aarch64/build/host-attr-2.4.48/.stamp_patched] Error 1
Traceback (most recent call last):
  File "./build-buildroot", line 181, in <module>
    Main().cli()
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 267, in cli
    exit_status = self.cli_noexit(*args, **kwargs)
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 258, in cli_noexit
    return self._do_main(vars(args))
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 1289, in main
    ret = self.timed_main()
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 1712, in timed_main
    return self.build()
  File "./build-buildroot", line 166, in build
    cwd=self.env['buildroot_source_dir'],
  File "/home/tom/git/linux-kernel-module-cheat/shell_helpers.py", line 403, in run_cmd
    raise e
Exception: Command exited with status: 2

I have no idea why this happens. Anyone can give me some help?

cirosantilli commented 5 years ago

I don't understand why, but buildroot is trying to reapply a patch that is already applied. I"m not sure what to try besides cleaning the out/buildroot/build/default/aarch64/build/attr or the entire build.

guoqinglei commented 5 years ago

I have updated buildroot to the latest commit of master branch, then rebuild using the command "./build --download-dependencies gem5-buildroot --buildroot-build-id brgcc8 --no-download-dependencie"

the error log is as follows:

/usr/bin/make -j1 O=/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64 HOSTCC="/usr/bin/gcc" HOSTCXX="/usr/bin/g++" syncconfig
make[1]: Entering directory '/home/tom/git/linux-kernel-module-cheat/submodules/buildroot'
  GEN     /home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64/Makefile
/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64/build/buildroot-config/conf: unrecognized option '--syncconfig'
Usage: /home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64/build/buildroot-config/conf [option] <kconfig-file>
[option] is _one_ of the following:
  --listnewconfig         List new options
  --oldaskconfig          Start a new configuration using a line-oriented program
  --oldconfig             Update a configuration using a provided .config as base
  --silentoldconfig       Same as oldconfig, but quietly, additionally update deps
  --olddefconfig          Same as silentoldconfig but sets new symbols to their default value
  --oldnoconfig           An alias of olddefconfig
  --defconfig <file>      New config with default defined in <file>
  --savedefconfig <file>  Save the minimal current configuration to <file>
  --allnoconfig           New config where all options are answered with no
  --allyesconfig          New config where all options are answered with yes
  --allmodconfig          New config where all options are answered with mod
  --alldefconfig          New config with all symbols set to default
  --randconfig            New config with random answer to all options
Makefile:989: recipe for target 'syncconfig' failed
make[1]: *** [syncconfig] Error 1
make[1]: Leaving directory '/home/tom/git/linux-kernel-module-cheat/submodules/buildroot'
Makefile:585: recipe for target '/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64/build/buildroot-config/auto.conf' failed
make: *** [/home/tom/git/linux-kernel-module-cheat/out/buildroot/build/brgcc8/aarch64/build/buildroot-config/auto.conf] Error 2
Traceback (most recent call last):
  File "./build", line 608, in <module>
    Main().cli()
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 267, in cli
    exit_status = self.cli_noexit(*args, **kwargs)
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 258, in cli_noexit
    return self._do_main(vars(args))
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 1289, in main
    ret = self.timed_main()
  File "./build", line 605, in timed_main
    component.build(self.env['arch'])
  File "./build", line 56, in build
    self.build_callback()
  File "./build", line 414, in f
    lkmc.import_path.import_path_main(component_file)(**args)
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 643, in __call__
    return super().__call__(**kwargs)
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 149, in __call__
    return self._do_main(kwargs)
  File "/home/tom/git/linux-kernel-module-cheat/cli_function.py", line 152, in _do_main
    return self.main(**self._get_args(kwargs))
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 1289, in main
    ret = self.timed_main()
  File "/home/tom/git/linux-kernel-module-cheat/common.py", line 1712, in timed_main
    return self.build()
  File "/home/tom/git/linux-kernel-module-cheat/build-buildroot", line 166, in build
    cwd=self.env['buildroot_source_dir'],
  File "/home/tom/git/linux-kernel-module-cheat/shell_helpers.py", line 403, in run_cmd
    raise e
Exception: Command exited with status: 2

I do not know how to fix it. Anyone knows ?

cirosantilli commented 5 years ago

Ah, I forgot about --buildroot-build-id, good idea to use it.

I'm afraid I don't know the solution, but once you do a ./build-buildroot --verbose &>build.log build, Buildroot will print every single command it takes, and it should not be hard to analyze why that is failing.

In this case I would start git grep and git log -p on the gcc tree to understand what happened to --syncconfig.

guoqinglei commented 5 years ago

In order to compile the program with SVE support, I have update the submodule/buildroot to the 2018.11 version, of which the gcc toolchain is 8.x default. On my Ubuntu 18.04 system, I have installed the gcc 8.3 version and set it the default. Then I tried following command: ./build_without_update_submodule_buildroot gem5-buildroot --buildroot-build-id 2018.11 --download-dependencies But I still saw that the ./out/buildroot/build/2018.11/aarch64/host/bin/aarch64-buildroot-linux-gnu-gcc --version is version 7.3. I do not know why this happens. I expected the aarch64-buildroot-linux-gnu-gcc version is 8.3.

There is a gcc directory in submodule/gcc, I wonder whether I should checkout this gcc to the 8.3 version ?

cirosantilli commented 5 years ago

Hi there,

Buildroot builds its own GCC, and I set it to use the source at submodules/gcc.

So yes, to use a different GCC through the Python scripts in this repo, you need to update that GCC.

But you could also just manually compile with the Ubuntu 8.3 GCC, and drop statically compiled executables into rootfs_overlay, not guaranteed to work but likely.

guoqinglei commented 5 years ago

I have tried to drop the compiled executable (eg. daxpy.o) into the rootfs like this:

cp daxpy.o /home/tom/git/linux-kernel-module-cheat/out/buildroot/build/default/aarch64/target

and then I run gem5:

./run --eval-after '/sample_package.out'

after that I run ./gem5.sh and enter into the root file system:

root@buildroot# ls
README.adoc         lkmc                sample_package.out
bin                 lost+found          sbin
dev                 media               sys
etc                 mnt                 tmp
home                opt                 usr
lib                 proc                var
lib64               root
linuxrc             run
/

There is not daxpy.o file at all. Why this happens ?

cirosantilli commented 5 years ago

Did you regenerate the image with ./build-buildroot?

Also, you should not drop stuff directly into buildroot dirs, maybe buildroot is cleaning up them. Use rootfs_overlay/ instead: https://cirosantilli.com/linux-kernel-module-cheat/#rootfs_overlay

guoqinglei commented 5 years ago

Did you regenerate the image with ./build-buildroot?

Also, you should not drop stuff directly into buildroot dirs, maybe buildroot is cleaning up them. Use rootfs_overlay/ instead: https://cirosantilli.com/linux-kernel-module-cheat/#rootfs_overlay

Not yet. I will try it now. Thanks.

cirosantilli commented 5 years ago

I have added a concrete SVE example at: https://cirosantilli.com/linux-kernel-module-cheat/#update-gcc-gcc-supported-by-buildroot at LKMC 6936bd6ba996dee40f7cd826e5cf01ef39c2cabf Will close for now, let me know if you can't reproduce.

cirosantilli commented 4 years ago

Also I've nowo updated Buildroot to 2019.11 which has GCC 8 by default and GCC 9 as an option: https://github.com/cirosantilli/linux-kernel-module-cheat/commit/ae17a79bba37007198505b55bf0f4accc02042af

guoqinglei commented 4 years ago

Thanks very much. Maybe I will have a try when I am available.

cirosantilli / linux-kernel-module-cheat

How to build applications with ARM SVE support using buildroot toolchains? #87