Use zstd for linux kernel modules

aviallon commented 3 months ago

Describe the bug

Because trust in XZ has sharply declined (https://github.com/NixOS/nixpkgs/issues/300055), and because Zstandard has comparable compression ratios and (much) faster (de)compression speed.

Steps To Reproduce

[user@system:~]$ zgrep 'XZ\|ZSTD' /proc/config.gz
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_ZSTD=n
CONFIG_KERNEL_XZ=y
# CONFIG_KERNEL_ZSTD is not set
CONFIG_RD_XZ=y
CONFIG_RD_ZSTD=y
CONFIG_MODULE_COMPRESS_XZ=y
# CONFIG_MODULE_COMPRESS_ZSTD is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
CONFIG_FW_LOADER_COMPRESS_XZ=y
# CONFIG_FW_LOADER_COMPRESS_ZSTD is not set
# CONFIG_ZRAM_DEF_COMP_ZSTD is not set
CONFIG_F2FS_FS_ZSTD=y
CONFIG_UBIFS_FS_ZSTD=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_ZSTD=y
CONFIG_CRYPTO_ZSTD=y
CONFIG_ZSTD_COMMON=y
CONFIG_ZSTD_COMPRESS=y
CONFIG_ZSTD_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
# CONFIG_XZ_DEC_MICROLZMA is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_ZSTD=y

Expected behavior

[user@system:~]$ zgrep 'XZ\|ZSTD' /proc/config.gz 
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_ZSTD=y
# CONFIG_KERNEL_XZ is not set
CONFIG_KERNEL_ZSTD=y
CONFIG_RD_XZ=y
CONFIG_RD_ZSTD=y
# CONFIG_MODULE_COMPRESS_XZ is not set
CONFIG_MODULE_COMPRESS_ZSTD=y
CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD=y
CONFIG_FW_LOADER_COMPRESS_XZ=y
CONFIG_FW_LOADER_COMPRESS_ZSTD=y
CONFIG_ZRAM_DEF_COMP_ZSTD=y
CONFIG_F2FS_FS_ZSTD=y
CONFIG_UBIFS_FS_ZSTD=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_ZSTD=y
CONFIG_CRYPTO_ZSTD=y
CONFIG_ZSTD_COMMON=y
CONFIG_ZSTD_COMPRESS=y
CONFIG_ZSTD_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
# CONFIG_XZ_DEC_MICROLZMA is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_ZSTD=y

Additional context

(https://github.com/NixOS/nixpkgs/issues/300055)

Notify maintainers

@TredwellGit @Ma27 @NeQuissimus @alyssais

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.7-xanmod1, NixOS, 23.11 (Tapir), 23.11.20240308.2be119a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `""`
 - nixpkgs: `/nix/store/y0c95bwyvs80pm69hdd4b11pyq2ghiwh-source`

Add a :+1: reaction to issues you find important.

thefossguy commented 3 months ago

Support for zstd has already been added to nixpkgs: #107703. All that remain is a similar change for the MODULE_COMPRESS_XZ config option: https://github.com/NixOS/nixpkgs/blob/d8489497c31f3b08eea1487466004f1f00c42668/pkgs/os-specific/linux/kernel/common-config.nix#L948

Ma27 commented 3 months ago

I'm aware of that and I already have a patch locally. Just didn't get around to push it yesterday.

thefossguy commented 3 months ago

Oh, I already pushed #302291. Hope you don't mind. 😅

Ma27 commented 3 months ago

Opened https://github.com/NixOS/nixpkgs/pull/302300 now. Your PR breaks at least 5.10 since the module-loading stuff was introduced in 5.13. Also, the issue suggested to add enable zstd for zram compression and the firmware loader which the other PR doesn't do.

thefossguy commented 3 months ago

Ack, will close #302291.

FlyingWombat commented 3 months ago

I think for the same reasons (mainly performance), we should also enable zstd support for linux-firmware #267442

ehmry commented 2 months ago

Because trust in XZ has sharply declined

Wrong. No one is claiming there are newly discovered problems in the XZ or LZMA algorithms. Please don't spread unfounded FUD and gossip.

aviallon commented 2 months ago

Because trust in XZ has sharply declined

Wrong. No one is claiming there are newly discovered problems in the XZ or LZMA algorithms. Please don't spread unfounded FUD and gossip.

I didn't say that XZ has any newly discovered problems. However, it is very much true that trust in the project has declined.

thefossguy commented 2 months ago

Trust has declined generally, primarily because a lot of codebases now need a thorough auditing. It's unfortunate that it happened to a project with a burnt out maintainer.

ehmry commented 2 months ago

Trust has declined generally, primarily because a lot of codebases now need a thorough auditing. It's unfortunate that it happened to a project with a burnt out maintainer.

Spreading FUD about projects being unmainted was how the malicous developer got commit access.

NixOS / nixpkgs