search

ClangBuiltLinux / linux

Linux kernel source tree
Other
241 stars 14 forks source link

Status: Building with Clang's integrated assembler aka LLVM_IAS=1 #1049

Closed dileks closed 4 years ago

dileks commented 4 years ago

@nickdesaulniers @nathanchance @tpimh

Hi,

what combination of recent Linux-kernel and recent llvm-toolchain is safe to build with enabled Clang's (I)ntegrated (AS)sembler aka set make-option LLVM_IAS=1?

kernel-doc llvm.rst [1] says:

Currently, the integrated assembler is disabled by default. You can pass LLVM_IAS=1 to enable it.

I am here on Debian/testing AMD64 with Linux v5.7.2 and llvm-toolchain-10 (release/10.x Git branch):

$ clang-10 --version
clang version 10.0.1 (https://github.com/llvm/llvm-project b6efa2365812f31667485c8948d49621ebf952f2)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/dileks/src/llvm-toolchain/install/bin

I have cherry-picked commit 51da9dfb7f20911ae4e79e9b412a9c2d4c373d4b from upstream:

$ git branch --show-current
for-5.7/elfnote-llvm-ias-ndesaulniers

$ git log --oneline v5.7.2..
604f98ca53d6 (HEAD -> for-5.7/elfnote-llvm-ias-ndesaulniers) elfnote: mark all .note sections SHF_ALLOC

Is this enough to have good chances for a successful build?

Thanks in advance.

Regards,

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/kbuild/llvm.rst#n62

dileks commented 4 years ago

Looks good with above patchset.

root@iniza:~# cat /proc/version 
Linux version 5.7.2-13-amd64-clang (sedat.dilek@gmail.com@iniza) (clang version 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2), LLD 11.0.0 (https://github.com/llvm/llvm-project 8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2)) #13~bullseye+dileks1 SMP 2020-06-16

The only warning I see in dmesg:

root@iniza:~# LC_ALL=C dmesg -T | egrep -i 'alg:|aes|crc32c|crypt' | egrep -i -v 'fscrypt|systemd'
[Tue Jun 16 12:15:24 2020] cryptd: max_cpu_qlen set to 1000
[Tue Jun 16 12:15:25 2020] AES CTR mode by8 optimization enabled
[Tue Jun 16 12:15:25 2020] alg: aead: rfc4106-gcm-aesni encryption test failed (wrong result) on test vector 0, cfg="two even aligned splits"
[Tue Jun 16 12:15:25 2020] alg: aead: generic-gcm-aesni encryption test failed (wrong result) on test vector 1, cfg="two even aligned splits"
[Tue Jun 16 12:15:25 2020] alg: No test for fips(ansi_cprng) (fips_ansi_cprng)
[Tue Jun 16 12:25:45 2020] Btrfs loaded, crc32c=crc32c-intel

Check Linux Git:

$ cd /path/to/linux/git
$ git grep -E 'generic-gcm-aesni|rfc4106-gcm-aesni' arch/x86/crypto/
arch/x86/crypto/aesni-intel_glue.c:             .cra_driver_name        = "__rfc4106-gcm-aesni",
arch/x86/crypto/aesni-intel_glue.c:             .cra_driver_name        = "__generic-gcm-aesni",

Loaded kernel-modules:

root@iniza:~# lsmod | egrep 'aes|crc32c|crypt' | sort
aesni_intel           368640  2
crc32c_generic         16384  0
crc32c_intel           24576  5
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
crypto_simd            16384  1 aesni_intel
glue_helper            16384  1 aesni_intel
libaes                 20480  2 bluetooth,aesni_intel
libcrc32c              16384  2 btrfs,xfs

The problems when re-running update-initramfs are gone with switching to tc-build/GOOD_REVISION.

dileks commented 4 years ago

For Linux v5.7.3-rc2 I switched back to make CC=clang-11 HOSTCC=clang-11 LD=ld.lld-11 HOSTLD=ld.lld-11 OBJDUMP=llvm-objdump LLVM_IAS=1 and dropped LLVM=1 and correspondig patches:

init: initramfs: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1
kernel: elfcore: Build with -no-integrated-as when LLVM=1 and LLVM_IAS=1
dileks commented 4 years ago

Just want to point out when you build with LLVM_IAS=1 and have forced DWARF version 4 assembler-option in the toplevel Makefile and do workaround with -no-integrated-as like for known broken aes_ctrby8_avx-x86_64.o on my system you assemble with GNU/as and DWARF version 2 is produced:

# Set CONFIG_DEBUG_INFO_DWARF4=y
cd /path/to/linux/git
scripts/config -e DEBUG_INFO_DWARF4

[ Makefile ]

ifdef CONFIG_DEBUG_INFO
ifdef CONFIG_DEBUG_INFO_SPLIT
DEBUG_CFLAGS    += -gsplit-dwarf
else
DEBUG_CFLAGS    += -g
endif
ifndef LLVM_IAS
KBUILD_AFLAGS   += -Wa,-gdwarf-2
endif
endif
ifdef CONFIG_DEBUG_INFO_DWARF4
DEBUG_CFLAGS    += -gdwarf-4
ifdef LLVM_IAS
KBUILD_AFLAGS  += -Wa,-gdwarf-4
endif
endif

[ arch/x86/crypto/Makefile ]

obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
ifdef LLVM_IAS
AFLAGS_aes_ctrby8_avx-x86_64.o += -no-integrated-as
endif

$ llvm-dwarfdump arch/x86/crypto/aes_ctrby8_avx-x86_64.o
arch/x86/crypto/aes_ctrby8_avx-x86_64.o:        file format elf64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x0000002a version = 0x0002 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x0000002e)

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_low_pc      (0x0000000000000000)
              DW_AT_high_pc     (0x0000000000003cb2)
              DW_AT_name        ("arch/x86/crypto/aes_ctrby8_avx-x86_64.S")
              DW_AT_comp_dir    ("/home/dileks/src/linux-kernel/git")
              DW_AT_producer    ("GNU AS 2.34")
              DW_AT_language    (DW_LANG_Mips_Assembler)

$ llvm-dwarfdump arch/x86/crypto/aesni-intel_asm.o | head -15
arch/x86/crypto/aesni-intel_asm.o:      file format elf64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x00002ad3 version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x00002ad7)

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_ranges      (0x00000000
                 [0x0000000000000000, 0x000000000000891f)
                 [0x0000000000000000, 0x000000000000003c))
              DW_AT_name        ("arch/x86/crypto/aesni-intel_asm.S")
              DW_AT_comp_dir    ("/home/dileks/src/linux-kernel/git")
              DW_AT_producer    ("clang version 11.0.0 (https://github.com/llvm/llvm-project 8da5b9083691b557f50f72ab099598bb291aec5f)")
              DW_AT_language    (DW_LANG_Mips_Assembler)

NOTE: See version = 0x0002 VS. version = 0x0004

UPDATE: 2020-06-18: Add Makefile excerpt and CONFIG_DEBUG_INFO_DWARF4=y

dileks commented 4 years ago

Linux v5.7.3 released - rebuild with latest patchset.

dileks commented 4 years ago

Progress on https://github.com/ClangBuiltLinux/linux/issues/1008#issuecomment-647423393

This means I do not need any -no-integrated-as workaround to build/assemble with LLVM_IAS=1.

dileks commented 4 years ago

Patch submitted for #1008

[1] https://github.com/ClangBuiltLinux/linux/issues/1008#issuecomment-647872808

dileks commented 4 years ago

Patch submitted for #1050

[1] https://github.com/ClangBuiltLinux/linux/issues/1050#issuecomment-647879758

dileks commented 4 years ago

I have re-tested with LLVM v10.0.1-rc2 and Linux v5.7.6 and LLVM_IAS=1.

This needs the no-integrated-as workarounds for memcpy_64 and memset_64.

Furthermore, when I upgrade my initrd-image with ZSTD support I still see the kernel double-faults.

Conclusion: With Linux v5.7.7 I will switch back to LLVM v11.0.1-git. (See tc-build: GOOD_REVISION=8a5aea7b50429cd4a459511286a7a9f1a7f4f5e2)

P.S.: With dropping my DWARF-4 assembler option patch I lose the information in the .o files (.S file as a base) when checking with llvm-dwarfdump.

dileks commented 4 years ago

This (last) Monday with the official release of Linux v5.8-rc4 I switched over from Linux v5.7.y.

Inspired by @paulmenzel report in #1076 I switched over my llvm-toolchain to version 1:11~++20200701093119+ffee8040534-1~exp1 from Debian/experimental.

We have accepted two patches from @jcai19 (see #1008) and @dileks (see #1050) in crypto-2.6.git:

"crypto: aesni - add compatibility with IAS"
"crypto: aesni - Fix build with LLVM_IAS=1"

The last patch to be upstreamed and be full compliant with LLVM_IAS=1 is the patch from @arndb (see #1010).

This means we are (will be) able to set and use on x86-64:

CONFIG_CRYPTO_AES_NI_INTEL={y,m}
CONFIG_CRYPTO_CRC32C_INTEL={y,m}

Just as a sidenote: I build and boot on bare metail with a manually setup LLVM=1 means with LLVM/utilities from Debian/experimental.

Note: One workaround I have with -no-integrated-as is for arch/x86/entry/entry_64.S (@jcai19 offered a patch in #1043). Feel free to test and report - I just compile/assemble-tested.

[1] https://github.com/ClangBuiltLinux/linux/issues/1076 [2] https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/commit/?id=44069737ac9625a0f02f0f7f5ab96aae4cd819bc [3] https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/commit/?id=3347c8a079d67af21760a78cc5f2abbcf06d9571 [4] https://lore.kernel.org/all/CA+icZUWQVrph_Rv9O=Q4Fh8Y0jXzkWGCHN5DFAaCZeE5eXoPbw@mail.gmail.com/ [5] https://lore.kernel.org/all/CA+icZUXj4TKEduA5A0fpdvha6E2fV6aZXOZ8BXvD5G4eYBHqsg@mail.gmail.com/ [6] https://llvm.org/pr24494 [7] https://github.com/ClangBuiltLinux/linux/issues/1043

dileks commented 4 years ago

The workaround for arch/x86/entry/entry_64.o is no more needed with the diff https://github.com/ClangBuiltLinux/linux/issues/1043#issuecomment-642163507 from @jcai19 and my diff https://github.com/ClangBuiltLinux/linux/issues/1088#issuecomment-657250733.

So a complete build with LLVM_IAS=1 on Linux v5.8-rc4 is OK.

dileks commented 4 years ago

To build Linux version 5.8 with LLVM toolchain version 11.0.0-rc1 and LLVM_IAS=1 (and LLVM=1) I have these patches applied:

  1. kbuild: Silence dwarf-2 warning when LLVM_IAS=1 Original-by: tpimh Link: https://github.com/ClangBuiltLinux/continuous-integration/blob/master/patches/llvm-all/linux-next/arm64/silence-dwarf2-warnings.patch
  2. kbuild: Add dwarf-4 assembler option when LLVM_IAS=1 Link: https://github.com/ClangBuiltLinux/linux/issues/1049#issuecomment-643746480
  3. crypto: aesni - add compatibility with IAS Link: https://git.kernel.org/linus/44069737ac9625a0f02f0f7f5ab96aae4cd819bc
  4. crypto: aesni - Fix build with LLVM_IAS=1 Link: https://git.kernel.org/linus/3347c8a079d67af21760a78cc5f2abbcf06d9571
  5. crypto: x86/crc32c - fix building with clang ias Link: https://git.kernel.org/linus/44623b2818f4a442726639572f44fd9b6d0ef68c
  6. x86: work around clang IAS bug referencing __force_order Link: https://lore.kernel.org/patchwork/patch/1248382/

NOTE to 1. and 2.: DWARF-4 seems to be a good choice (sidenote: binutils >= 2.35 has support for it)

NOTE to 2.: CONFIG_DEBUG_INFO_DWARF4=y

NOTE to 3. and 4.: CONFIG_CRYPTO_AES_NI_INTEL=m

NOTE to 5.: CONFIG_CRYPTO_CRC32C_INTEL=m

NOTE to 6.: Required for some kernel-modules like out-of-tree VirtualBox or CONFIG_LKDTM=m (see #1120).

Update 16-Aug-2020: Replaced patches both DWARF patches by... Introduce CONFIG_DEBUG_INFO_AS_DWARF2 and CONFIG_DEBUG_INFO_AS_DWARF4 Link: https://patchwork.kernel.org/patch/11716107/

Update 19-Aug-2020: All x86/crypto patches now in Linux v5.8.2 (and other Linux-stable releases)

Update 05-Aug-2020: All x86/crypto patches now in Linux 5.9

Update 03-Aug-2020: Linux version 5.8 boots fine, see https://github.com/ClangBuiltLinux/linux/issues/1117#issuecomment-667797539

Update 01-Aug-2020: Add @arndb x86: work around clang IAS bug referencing __force_order patch to be able to build VirtualBox out-of-tree kernel-module (see https://github.com/ClangBuiltLinux/linux/issues/1104#issuecomment-667470053)

Update 23-Jul-2020: All x86/crypto patches now in https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/.

Update 22-Jul-2020: Initial version

dileks commented 4 years ago

Update: Linux v5.9-rc1:

$ cat /proc/version 
Linux version 5.9.0-rc1-3-amd64-llvm11-ias (sedat.dilek@gmail.com@iniza) (clang version 11.0.0 (https://github.com/llvm/llvm-project c2f52e2c1288b00eae528825fb92668b1f3df732), LLD 11.0.0 (https://github.com/llvm/llvm-project c2f52e2c1288b00eae528825fb92668b1f3df732)) #3~bullseye+dileks1 SMP 2020-08-18

Required patches/patchsets:

  1. x86: work around clang IAS bug referencing __force_order Link: https://lore.kernel.org/patchwork/patch/1248382/ Fixes: https://github.com/ClangBuiltLinux/linux/issues/1104#issuecomment-667470053 Fixes: https://github.com/ClangBuiltLinux/linux/issues/1120
  2. x86/boot: Remove run-time relocations from compressed kernel v6 Link: https://lore.kernel.org/patchwork/project/lkml/list/?series=456251 Git: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/boot Fixes: https://github.com/ClangBuiltLinux/linux/issues/1129

Optional DWARF version 4:

  1. Introduce CONFIG_DEBUG_INFO_AS_DWARF2 and CONFIG_DEBUG_INFO_AS_DWARF4 (see https://patchwork.kernel.org/patch/11716107/)
  2. kbuild: Simplify DEBUG_INFO Kconfig handling (see https://patchwork.kernel.org/patch/11716109/)

Perf stat:

 Performance counter stats for 'make V=1 -j3 HOSTCC=clang-11 HOSTCXX=clang++-11 HOSTLD=ld.lld-11 HOSTAR=llvm-ar-11 CC=clang-11 LD=ld.lld-11 AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11 LLVM_IAS=1 LOCALVERSION=-3-amd64-llvm11-ias KBUILD_VERBOSE=1 KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@gmail.com KBUILD_BUILD_TIMESTAMP=2020-08-18 bindeb-pkg KDEB_PKGVERSION=5.9.0~rc1-3~bullseye+dileks1':

       41454490.82 msec task-clock:u              #    2.896 CPUs utilized          
                 0      context-switches:u        #    0.000 K/sec                  
                 0      cpu-migrations:u          #    0.000 K/sec                  
         308569350      page-faults:u             #    0.007 M/sec                  
    75665754549051      cycles:u                  #    1.825 GHz                    
    60254260110478      stalled-cycles-frontend:u #   79.63% frontend cycles idle   
    48829161160508      stalled-cycles-backend:u  #   64.53% backend cycles idle    
    48186232262620      instructions:u            #    0.64  insn per cycle         
                                                  #    1.25  stalled cycles per insn
     9222134517271      branches:u                #  222.464 M/sec                  
      389598075528      branch-misses:u           #    4.22% of all branches        

   14311.995383874 seconds time elapsed

   39452.053992000 seconds user
    2058.547318000 seconds sys

Attachments: My kernel-config and dmesg-output

config-5.9.0-rc1-3-amd64-llvm11-ias.txt dmesg-T_5.9.0-rc1-3-amd64-llvm11-ias.txt

nickdesaulniers commented 4 years ago

please file individual bug reports for specific failures specific to either a missing feature or particular translation unit.

dileks commented 4 years ago

@nickdesaulniers

Clang-IAS is fine with LLVM toolchain version 11 rc2+ and recent Linux-kernel version like 5.8 (with some upstream fixes) or 5.9-rcX.

Currently, I use Linux version 5.9-rc5 and LLVM toolchain version 11.0.0 (https://github.com/llvm/llvm-project 97ac9e82002d6b12831ca2c78f739cca65a4fa05) (ThinLTO and PGO optimized) with no issues.

What is the status with LLVM toolchain version 10.0.1 (which will be default with Linux version 5.10)? Which Linux-kernel version supports fully Clang-IAS? Do we have all Clang-IAS related patches in all backports (Linux and LLVM toolchain)?

As you better know Clang-IAS is prerequisite for Clang-LTO and Clang-CFI.

I left this issue open that people have an orientation. If you have a better solution or maybe there is a link I do not know off?

nickdesaulniers commented 4 years ago

Which Linux-kernel version supports fully Clang-IAS?

None. LLVM_IAS=1 is experimental/WIP for now.

If you have a better solution or maybe there is a link I do not know off?

Either the label "integrated-as" or maybe a github "milestone" might be more appropriate?