search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.61k stars 1.75k forks source link

In-tree ZFS-ZSTD clash with in-kernel ZSTD #10763

Closed nivedita76 closed 4 years ago

nivedita76 commented 4 years ago

System information

Type Version/Name
Distribution Name
Distribution Version
Linux Kernel
Architecture
ZFS Version
SPL Version

Describe the problem you're observing

ZSTD is already in the linux kernel, and it produces multiple definition errors if ZSTD is builtin to the kernel and ZFS is being compiled as builtin.

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

thehaven commented 4 years ago

Seeing this also against both linux-5.8.0-gentoo-r1 and gentoo-sources-5.8.2, example below is from linux-5.8.0-gentoo-r1:

saratoga /usr/src/linux # grep ZSTD .config
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
# CONFIG_SQUASHFS_ZSTD is not set
# CONFIG_PSTORE_ZSTD_COMPRESS is not set
# CONFIG_CRYPTO_ZSTD is not set
CONFIG_ZSTD_COMPRESS=y
CONFIG_ZSTD_DECOMPRESS=y
saratoga /usr/src/linux # make -j 81
scripts/kconfig/conf  --syncconfig Kconfig
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GZIP    kernel/config_data.gz
  CC      kernel/configs.o
Cyclomatic Complexity 1 kernel/configs.c:ikconfig_cleanup
Cyclomatic Complexity 2 kernel/configs.c:ikconfig_init
Cyclomatic Complexity 1 kernel/configs.c:ikconfig_read_current
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
ld: fs/btrfs/zstd.o: in function `zstd_decompress':
/usr/src/linux/fs/btrfs/zstd.c:627: multiple definition of `zstd_decompress'; fs/zfs/zstd/zfs_zstd.o:/usr/src/linux/fs/zfs/zstd/zfs_zstd.c:526: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_buildCTable_wksp':
/usr/src/linux/lib/zstd/fse_compress.c:93: multiple definition of `FSE_buildCTable_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7548: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_NCountWriteBound':
/usr/src/linux/lib/zstd/fse_compress.c:200: multiple definition of `FSE_NCountWriteBound'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7668: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_writeNCount':
/usr/src/linux/lib/zstd/fse_compress.c:303: multiple definition of `FSE_writeNCount'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7769: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_optimalTableLog_internal':
/usr/src/linux/lib/zstd/fse_compress.c:495: multiple definition of `FSE_optimalTableLog_internal'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7806: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_optimalTableLog_internal':
/usr/src/linux/lib/zstd/fse_compress.c:495: multiple definition of `FSE_optimalTableLog'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7819: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_normalizeCount':
/usr/src/linux/lib/zstd/fse_compress.c:609: multiple definition of `FSE_normalizeCount'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7917: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_buildCTable_raw':
/usr/src/linux/lib/zstd/fse_compress.c:667: multiple definition of `FSE_buildCTable_raw'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7978: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_buildCTable_rle':
/usr/src/linux/lib/zstd/fse_compress.c:710: multiple definition of `FSE_buildCTable_rle'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8018: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_compress_usingCTable':
/usr/src/linux/lib/zstd/fse_compress.c:787: multiple definition of `FSE_compress_usingCTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8096: first defined here
ld: lib/zstd/fse_compress.o: in function `FSE_compressBound':
/usr/src/linux/lib/zstd/fse_compress.c:795: multiple definition of `FSE_compressBound'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8105: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_optimalTableLog':
/usr/src/linux/lib/zstd/huf_compress.c:70: multiple definition of `HUF_optimalTableLog'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8413: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_buildCTable_wksp':
/usr/src/linux/lib/zstd/huf_compress.c:421: multiple definition of `HUF_buildCTable_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8703: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compressBound':
/usr/src/linux/lib/zstd/huf_compress.c:526: multiple definition of `HUF_compressBound'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8805: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress1X_usingCTable':
/usr/src/linux/lib/zstd/huf_compress.c:548: multiple definition of `HUF_compress1X_usingCTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8894: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress4X_usingCTable':
/usr/src/linux/lib/zstd/huf_compress.c:592: multiple definition of `HUF_compress4X_usingCTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8929: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress4X_usingCTable':
/usr/src/linux/lib/zstd/huf_compress.c:592: multiple definition of `HUF_compress4X_usingCTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:8929: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress1X_wksp':
/usr/src/linux/lib/zstd/huf_compress.c:750: multiple definition of `HUF_compress1X_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:9096: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress1X_repeat':
/usr/src/linux/lib/zstd/huf_compress.c:757: multiple definition of `HUF_compress1X_repeat'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:9108: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress4X_wksp':
/usr/src/linux/lib/zstd/huf_compress.c:763: multiple definition of `HUF_compress4X_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:9130: first defined here
ld: lib/zstd/huf_compress.o: in function `HUF_compress4X_repeat':
/usr/src/linux/lib/zstd/huf_compress.c:770: multiple definition of `HUF_compress4X_repeat'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:9145: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressContinue':
/usr/src/linux/lib/zstd/compress.c:2541: multiple definition of `ZSTD_compressContinue'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:15881: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBlock_greedy_extDict':
/usr/src/linux/lib/zstd/compress.c:2252: multiple definition of `ZSTD_compressBlock_greedy_extDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:19497: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_seqToCodes':
/usr/src/linux/lib/zstd/compress.c:567: multiple definition of `ZSTD_seqToCodes'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:15014: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_initCStream_usingCDict':
/usr/src/linux/lib/zstd/compress.c:3106: multiple definition of `ZSTD_initCStream_usingCDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16811: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_flushStream':
/usr/src/linux/lib/zstd/compress.c:3239: multiple definition of `ZSTD_flushStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17175: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compress_usingCDict':
/usr/src/linux/lib/zstd/compress.c:2931: multiple definition of `ZSTD_compress_usingCDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16684: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBegin_usingCDict':
/usr/src/linux/lib/zstd/compress.c:2915: multiple definition of `ZSTD_compressBegin_usingCDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16660: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compress_usingDict':
/usr/src/linux/lib/zstd/compress.c:2827: multiple definition of `ZSTD_compress_usingDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16373: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_resetCStream':
/usr/src/linux/lib/zstd/compress.c:3050: multiple definition of `ZSTD_resetCStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:14050: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_adjustCParams':
/usr/src/linux/lib/zstd/compress.c:182: multiple definition of `ZSTD_adjustCParams'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:14152: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_copyCCtx':
/usr/src/linux/lib/zstd/compress.c:350: multiple definition of `ZSTD_copyCCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:14919: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_freeCStream':
/usr/src/linux/lib/zstd/compress.c:3004: multiple definition of `ZSTD_freeCStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:13233: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressStream':
/usr/src/linux/lib/zstd/compress.c:3224: multiple definition of `ZSTD_compressStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17054: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBegin':
/usr/src/linux/lib/zstd/compress.c:2760: multiple definition of `ZSTD_compressBegin'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16249: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_freeCDict':
/usr/src/linux/lib/zstd/compress.c:2901: multiple definition of `ZSTD_freeCDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16552: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBegin_advanced':
/usr/src/linux/lib/zstd/compress.c:2748: multiple definition of `ZSTD_compressBegin_advanced'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16229: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_getCParams':
/usr/src/linux/lib/zstd/compress.c:3414: multiple definition of `ZSTD_getCParams'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17337: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBegin_usingDict':
/usr/src/linux/lib/zstd/compress.c:2755: multiple definition of `ZSTD_compressBegin_usingDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16239: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBound':
/usr/src/linux/lib/zstd/compress.c:38: multiple definition of `ZSTD_compressBound'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:13129: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_createCStream_advanced':
/usr/src/linux/lib/zstd/compress.c:2983: multiple definition of `ZSTD_createCStream_advanced'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16707: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_checkCParams':
/usr/src/linux/lib/zstd/compress.c:155: multiple definition of `ZSTD_checkCParams'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:10272: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_maxCLevel':
/usr/src/linux/lib/zstd/compress.c:3295: multiple definition of `ZSTD_maxCLevel'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17200: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_getSeqStore':
/usr/src/linux/lib/zstd/compress.c:141: multiple definition of `ZSTD_getSeqStore'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:13274: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_endStream':
/usr/src/linux/lib/zstd/compress.c:3252: multiple definition of `ZSTD_endStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17182: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBlock':
/usr/src/linux/lib/zstd/compress.c:2547: multiple definition of `ZSTD_compressBlock'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:15893: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_CStreamInSize':
/usr/src/linux/lib/zstd/compress.c:3023: multiple definition of `ZSTD_CStreamInSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16720: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_initCStream':
/usr/src/linux/lib/zstd/compress.c:3093: multiple definition of `ZSTD_initCStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16866: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_freeCCtx':
/usr/src/linux/lib/zstd/compress.c:134: multiple definition of `ZSTD_freeCCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:13233: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_getParams':
/usr/src/linux/lib/zstd/compress.c:3438: multiple definition of `ZSTD_getParams'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:17359: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressBound':
/usr/src/linux/lib/zstd/compress.c:38: multiple definition of `ZSTD_CStreamOutSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16725: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressCCtx':
/usr/src/linux/lib/zstd/compress.c:2832: multiple definition of `ZSTD_compressCCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16388: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_invalidateRepCodes':
/usr/src/linux/lib/zstd/compress.c:341: multiple definition of `ZSTD_invalidateRepCodes'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:14679: first defined here
ld: lib/zstd/compress.o: in function `ZSTD_compressEnd':
/usr/src/linux/lib/zstd/compress.c:2807: multiple definition of `ZSTD_compressEnd'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:16299: first defined here
ld: lib/zstd/entropy_common.o: in function `FSE_versionNumber':
/usr/src/linux/lib/zstd/entropy_common.c:49: multiple definition of `FSE_versionNumber'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2504: first defined here
ld: lib/zstd/entropy_common.o: in function `ERR_isError':
/usr/src/linux/lib/zstd/error_private.h:44: multiple definition of `FSE_isError'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:810: first defined here
ld: lib/zstd/entropy_common.o: in function `HUF_isError':
/usr/src/linux/lib/zstd/entropy_common.c:52: multiple definition of `HUF_isError'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2509: first defined here
ld: lib/zstd/entropy_common.o: in function `FSE_readNCount':
/usr/src/linux/lib/zstd/entropy_common.c:60: multiple definition of `FSE_readNCount'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2520: first defined here
ld: lib/zstd/fse_decompress.o: in function `FSE_buildDTable_rle':
/usr/src/linux/lib/zstd/fse_decompress.c:180: multiple definition of `FSE_buildDTable_rle'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2895: first defined here
ld: lib/zstd/fse_decompress.o: in function `FSE_buildDTable_raw':
/usr/src/linux/lib/zstd/fse_decompress.c:188: multiple definition of `FSE_buildDTable_raw'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2904: first defined here
ld: lib/zstd/fse_decompress.o: in function `FSE_decompress_usingDTable':
/usr/src/linux/lib/zstd/fse_decompress.c:283: multiple definition of `FSE_decompress_usingDTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:2995: first defined here
ld: lib/zstd/fse_decompress.o: in function `FSE_decompress_wksp':
/usr/src/linux/lib/zstd/fse_decompress.c:295: multiple definition of `FSE_decompress_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:3007: first defined here
ld: lib/zstd/zstd_common.o: in function `ZSTD_malloc':
/usr/src/linux/lib/zstd/zstd_common.c:69: multiple definition of `ZSTD_malloc'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7374: first defined here
ld: lib/zstd/zstd_common.o: in function `ZSTD_free':
/usr/src/linux/lib/zstd/zstd_common.c:73: multiple definition of `ZSTD_free'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:7395: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_readDTableX2_wksp':
/usr/src/linux/lib/zstd/huf_decompress.c:91: multiple definition of `HUF_readDTableX2_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:21903: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress1X2_usingDTable':
/usr/src/linux/lib/zstd/huf_decompress.c:227: multiple definition of `HUF_decompress1X2_usingDTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22222: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress1X2_DCtx_wksp':
/usr/src/linux/lib/zstd/huf_decompress.c:233: multiple definition of `HUF_decompress1X2_DCtx_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22229: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress4X2_usingDTable':
/usr/src/linux/lib/zstd/huf_decompress.c:358: multiple definition of `HUF_decompress4X2_usingDTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22262: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress4X2_DCtx_wksp':
/usr/src/linux/lib/zstd/huf_decompress.c:364: multiple definition of `HUF_decompress4X2_DCtx_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22284: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress1X_usingDTable':
/usr/src/linux/lib/zstd/huf_decompress.c:848: multiple definition of `HUF_decompress1X_usingDTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22324: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress4X_usingDTable':
/usr/src/linux/lib/zstd/huf_decompress.c:855: multiple definition of `HUF_decompress4X_usingDTable'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22343: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_selectDecoder':
/usr/src/linux/lib/zstd/huf_decompress.c:890: multiple definition of `HUF_selectDecoder'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22392: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress4X_hufOnly_wksp':
/usr/src/linux/lib/zstd/huf_decompress.c:927: multiple definition of `HUF_decompress4X_hufOnly_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22470: first defined here
ld: lib/zstd/huf_decompress.o: in function `HUF_decompress1X_DCtx_wksp':
/usr/src/linux/lib/zstd/huf_decompress.c:942: multiple definition of `HUF_decompress1X_DCtx_wksp'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:22495: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompressBlock':
/usr/src/linux/lib/zstd/decompress.c:1480: multiple definition of `ZSTD_decompressBlock'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:27819: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompressBegin_usingDict':
/usr/src/linux/lib/zstd/decompress.c:1970: multiple definition of `ZSTD_decompressBegin_usingDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25688: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_nextInputType':
/usr/src/linux/lib/zstd/decompress.c:1725: multiple definition of `ZSTD_nextInputType'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25367: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_DStreamOutSize':
/usr/src/linux/lib/zstd/decompress.c:2278: multiple definition of `ZSTD_DStreamOutSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25795: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_resetDStream':
/usr/src/linux/lib/zstd/decompress.c:2282: multiple definition of `ZSTD_resetDStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24597: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_getDictID_fromDict':
/usr/src/linux/lib/zstd/decompress.c:2108: multiple definition of `ZSTD_getDictID_fromDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25725: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompress_usingDDict':
/usr/src/linux/lib/zstd/decompress.c:2150: multiple definition of `ZSTD_decompress_usingDDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25761: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_findFrameCompressedSize':
/usr/src/linux/lib/zstd/decompress.c:1511: multiple definition of `ZSTD_findFrameCompressedSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25039: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_createDCtx_advanced':
/usr/src/linux/lib/zstd/decompress.c:130: multiple definition of `ZSTD_createDCtx_advanced'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24642: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_getcBlockSize':
/usr/src/linux/lib/zstd/decompress.c:396: multiple definition of `ZSTD_getcBlockSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:26452: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_freeDStream':
/usr/src/linux/lib/zstd/decompress.c:2258: multiple definition of `ZSTD_freeDStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25788: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_freeDCtx':
/usr/src/linux/lib/zstd/decompress.c:149: multiple definition of `ZSTD_freeDCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24668: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_nextSrcSizeToDecompress':
/usr/src/linux/lib/zstd/decompress.c:1721: multiple definition of `ZSTD_nextSrcSizeToDecompress'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25346: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_findDecompressedSize':
/usr/src/linux/lib/zstd/decompress.c:320: multiple definition of `ZSTD_findDecompressedSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24884: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_isFrame':
/usr/src/linux/lib/zstd/decompress.c:175: multiple definition of `ZSTD_isFrame'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24703: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decodeSeqHeaders':
/usr/src/linux/lib/zstd/decompress.c:795: multiple definition of `ZSTD_decodeSeqHeaders'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:26883: first defined here
ld: lib/zstd/decompress.o: in function `memcpy':
/usr/src/linux/./include/linux/string.h:406: multiple definition of `ZSTD_copyDCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/./include/linux/string.h:406: first defined here
ld: lib/zstd/decompress.o: in function `memcpy':
/usr/src/linux/./include/linux/string.h:406: multiple definition of `ZSTD_decompressBegin'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24597: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompressStream':
/usr/src/linux/lib/zstd/decompress.c:2299: multiple definition of `ZSTD_decompressStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:26095: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompressContinue':
/usr/src/linux/lib/zstd/decompress.c:1744: multiple definition of `ZSTD_decompressContinue'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25395: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_initDStream':
/usr/src/linux/lib/zstd/decompress.c:2215: multiple definition of `ZSTD_initDStream'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25850: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompress_usingDict':
/usr/src/linux/lib/zstd/decompress.c:1709: multiple definition of `ZSTD_decompressDCtx'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25320: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_getDictID_fromFrame':
/usr/src/linux/lib/zstd/decompress.c:2136: multiple definition of `ZSTD_getDictID_fromFrame'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24756: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decompress_usingDict':
/usr/src/linux/lib/zstd/decompress.c:1709: multiple definition of `ZSTD_decompress_usingDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25298: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_getDictID_fromDDict':
/usr/src/linux/lib/zstd/decompress.c:2121: multiple definition of `ZSTD_getDictID_fromDDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24442: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_insertBlock':
/usr/src/linux/lib/zstd/decompress.c:1491: multiple definition of `ZSTD_insertBlock'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25076: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_getFrameParams':
/usr/src/linux/lib/zstd/decompress.c:211: multiple definition of `ZSTD_getFrameContentSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24844: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_decodeLiteralsBlock':
/usr/src/linux/lib/zstd/decompress.c:434: multiple definition of `ZSTD_decodeLiteralsBlock'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:26474: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_freeDDict':
/usr/src/linux/lib/zstd/decompress.c:2091: multiple definition of `ZSTD_freeDDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:24414: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_initDStream_usingDDict':
/usr/src/linux/lib/zstd/decompress.c:2248: multiple definition of `ZSTD_initDStream_usingDDict'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25859: first defined here
ld: lib/zstd/decompress.o: in function `ZSTD_DStreamInSize':
/usr/src/linux/lib/zstd/decompress.c:2277: multiple definition of `ZSTD_DStreamInSize'; fs/zfs/zstd/lib/zstd.o:/usr/src/linux/fs/zfs/zstd/lib/zstd.c:25795: first defined here
make: *** [Makefile:1139: vmlinux] Error 1

I'm building against ZFS HEAD (commit 64025fa3a1f0f710f7f8678f2ac459b07ed9f88f) and suspect this is related to the merge yesterday of https://github.com/openzfs/zfs/pull/10278#issuecomment-677809493

PrivatePuffin commented 4 years ago

Thanks for the headsup! None of our testing systems had these issues...

@c0d3z3r0 and @brainslayer You guys might want to provide a fix...

Edit I was wrong we did test 5.8, thanks @brainslayer

BrainSlayer commented 4 years ago

mine is running 5.8 with current master git including zstd of course. all working

BrainSlayer commented 4 years ago

server2:~ # uname -a Linux server2 5.8.2-666.ddwrt #1 SMP Fri Aug 21 13:02:10 +03 2020 x86_64 x86_64 x86_64 GNU/Linux server2:~ # zfs get all |grep compres zfs compressratio 2.04x - zfs compression zstd local zfs refcompressratio 2.04x - server2:~ #

BrainSlayer commented 4 years ago

but he is doing inline kernel compiling. i'm compiling out of tree

BrainSlayer commented 4 years ago

ah the cause is easy. the problem is that he is compiling with zstd in kernel. this collides with symbols from zfs zstd implementation. this is unresolvable unless we rename all zstd functions in our implementation. blocking point

PrivatePuffin commented 4 years ago

@BrainSlayer I'm not that well versed into this, but can't we exclude the ZSTD included in the kernel on build or is it automatically included for everything without giving us the option not to?

BrainSlayer commented 4 years ago

no we cannot. the in kernel zstd implementation is feature reduced and out of date

nivedita76 commented 4 years ago

Why did we need our own implementation? ZSTD has been in the kernel for a while now I think. Can’t we just use it?

From: Sebastian Gottschall notifications@github.com Sent: Friday, August 21, 2020 11:23:16 AM To: openzfs/zfs zfs@noreply.github.com Cc: Arvind Sankar nivedita@alum.mit.edu; Author author@noreply.github.com Subject: Re: [openzfs/zfs] ZSTD clash with linux 5.8.x (#10763)

ah the cause is easy. the problem is that he is compiling with zstd in kernel support. this collides with symbols from zfs zstd implementation. this is unresolvable unless we rename all zstd functions in our implementation. blocking point

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/openzfs/zfs/issues/10763#issuecomment-678348711, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJBU6WIW2CQXKWFNHZ4KYMDSB2GOJANCNFSM4QG654KA.

PrivatePuffin commented 4 years ago

@nivedita76 We can't rely on the version in the kernel, because it might mismatch the version that originally compressed your data. Simply put: If you don't want you data be corrupted you don't want to rely on the kernel ZSTD.

TLDR: No.

BrainSlayer commented 4 years ago

Why did we need our own implementation? ZSTD has been in the kernel for a while now I think. Can’t we just use it? … ____ From: Sebastian Gottschall notifications@github.com Sent: Friday, August 21, 2020 11:23:16 AM To: openzfs/zfs zfs@noreply.github.com Cc: Arvind Sankar nivedita@alum.mit.edu; Author author@noreply.github.com Subject: Re: [openzfs/zfs] ZSTD clash with linux 5.8.x (#10763) ah the cause is easy. the problem is that he is compiling with zstd in kernel support. this collides with symbols from zfs zstd implementation. this is unresolvable unless we rename all zstd functions in our implementation. blocking point — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10763 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJBU6WIW2CQXKWFNHZ4KYMDSB2GOJANCNFSM4QG654KA.

as i wrote in last comment. the in kernel zstd lib is out of date and feature reduced. it will not work. but you can compile zfs out of tree as workaround until we agree how we fix that problem. we have only a symbol collision since you compiled it static

PrivatePuffin commented 4 years ago

@BrainSlayer Any idea how many Distro's would be hit with this issue out-of-the-box?

BrainSlayer commented 4 years ago

100%. The in kernel zstd lib is included since long time. so if you compile zfs static into the kernel and in kernel zstd is enabled, all supported kernels are affected. compiling out of tree is no issue

BrainSlayer commented 4 years ago

solution writing a wrapper header with some #define macros which rename all zstd functions for zfs

PrivatePuffin commented 4 years ago

@BrainSlayer I meant like: How many kernels compile zfs static into the kernel AND have the in-kernel zstd both enabled?

PrivatePuffin commented 4 years ago

solution writing a wrapper header with some #define macros which rename all zstd functions for zfs

We could indeed prefix everything with ZFS_, the priority for doing such a thing depends a bit how many distro's would be affected out-of-the-box by compiling zfs static into the kernel AND having the in-kernel zstd both enabled...

nivedita76 commented 4 years ago

@BrainSlayer what would happen currently if someone loads zfs module and then loads some other kernel module that uses in-kernel ZSTD?

For renaming, it's possible to do it using objcopy --prefix-symbols=zfs_ (for eg) to avoid having to list out all the symbols. An example in the kernel is drivers/firmware/efi/libstub/Makefile, which renames all symbols for the EFI stub on ARM64.

PrivatePuffin commented 4 years ago

@nivedita76 Could you rename the title to, please: In-tree ZFS-ZSTD clash with in-kernel ZSTD

For renaming, it's possible to do it using objcopy --prefix-symbols=zfs_ (for eg) to avoid having to list out all the symbols

I'm okey with such a solution personally. @allanjude and @c0d3z3r0 ?

nivedita76 commented 4 years ago

Regarding distros, the upstream kernel now supports initramfs compressed using ZSTD. Not sure how many distros have backported that, but ZSTD compiled into the kernel might be fairly common.

PrivatePuffin commented 4 years ago

@nivedita76 Thats what @BrainSlayer told me previously, but thats not the issue. The issue is having ZSTD "baked in" AND building ZFS in-tree. (the combination of both)

But yes, if building ZFS in-tree all kernel versions including ZSTD in-kernel would be affected.

nivedita76 commented 4 years ago

@nivedita76 Thats what @BrainSlayer told me previously, but thats not the issue. The issue is having ZSTD "baked in" AND building ZFS in-tree. (the combination of both)

Are we sure this doesn't cause runtime issues? The ZFS ZSTD implementation is loaded as a module, right? How can that override ZSTD symbols that are already built into the kernel?

nivedita76 commented 4 years ago

What I mean is, are we sure, zfs.ko is using the symbols from zzstd.ko, and not the ones built into the kernel?

PrivatePuffin commented 4 years ago

@nivedita76 I think we also tested it in Kernel versions without ZSTD...

But I think just prefixing our version is a nice and clean solution.

BrainSlayer commented 4 years ago

@Ornias1993 i agree. i will prepare something as soon as i find some time. there are alot of functions to consider

BrainSlayer commented 4 years ago

@nivedita76 zzstd.ko is using all functions inline. there is no reference to zstd from zfs.ko etc. so all references to zstd itself are in zzstd.ko itself. so there is no collision if you just compile it out of tree and not static into the kernel

PrivatePuffin commented 4 years ago

@Ornias1993 i agree. i will prepare something as soon as i find some time. there are alot of functions to consider

Would the solution from @nivedita76 work out for that purpose? objcopy --prefix-symbols=zfs_

nivedita76 commented 4 years ago

@nivedita76 I think we also tested it in Kernel versions without ZSTD...

But that doesn't address my concern, which is on kernel versions with ZSTD. Or really, trying to import pools created with kernel w/o ZSTD, which definitely used in-tree ZSTD, on a kernel with ZSTD, which might be using the in-kernel ZSTD.

nivedita76 commented 4 years ago

@nivedita76 zzstd.ko is using all functions inline. there is no reference to zstd from zfs.ko etc. so all references to zstd itself are in zzstd.ko itself. so there is no collision if you just compile it out of tree and not static into the kernel

Oh, not even zstd_compress/zstd_decompress collide?

PrivatePuffin commented 4 years ago

@nivedita76 If it did, we would've the same issue as we have here, wouldn't we? ;) Because if that was the case, It would also error out because it would've two options for zstd_compress/zstd_decompress...

nivedita76 commented 4 years ago

Ok, I wasn't sure if it errored on symbol clashes in modules vs kernel.

PrivatePuffin commented 4 years ago

@nivedita76 I'm not 100% sure, but if we fix this bug that would be fixed anyway if thats the case. It should at least be fixed on OpenZFS2.0 RELEASE :)

BrainSlayer commented 4 years ago

please just let me sleep this night. tomorrow we have a solution. its easy to fix and no there is no risc of using wrong symbols if its compiled out of tree as module

nivedita76 commented 4 years ago

lol @BrainSlayer sweet dreams :)

BrainSlayer commented 4 years ago

@Ornias1993 one interesting point. isnt static linking non gpl code like zfs to the kernel a gpl license violation?

BrainSlayer commented 4 years ago

@nivedita76 please try this commit/patch on top of master and give us some feedback if this solves your problem https://github.com/BrainSlayer/zfs/commit/2fee2750a68ed7826d17554181971c0ee6cb703b

PrivatePuffin commented 4 years ago

@BrainSlayer As long as you don't distribute or publish, thats not a GPL violation according to article 2 section b of the GPLv2.

Funny enough the GPL_ONLY tags are a GPL violation themselves, in accordance to article 6 of the GPLv2 and the general explaination of which in the Preamble which, in contrast to what a lot of people say, is a legally accepted source to explain articles of a licence or agreement in (at least) Europe.

TLDR:

  1. By static linking it yourself, you're not in violation
  2. The Linux kernel itself is the biggest GPL violator worldwide.
PrivatePuffin commented 4 years ago

@BrainSlayer Didnt the patch suggested by @nivedita76 work out?

BrainSlayer commented 4 years ago

@Ornias1993 havent seen a patch from him. i made my own (see my commit link to my private tree). just prefixing the symbols isnt enough since you also need to prefix the references to it which cannot easily done by objcopy, except with a symbol list. so my solution is more clean (usually)

regarding the linux kernel, i dont understand why you say that the kernel is a biggest violator by itself. explain

PrivatePuffin commented 4 years ago

@BrainSlayer Ohh okey, I was refering to his patch suggestion. Good point indeed.

I explained why Linux is the biggest violator already, second part of my last post.

PrivatePuffin commented 4 years ago

@BrainSlayer Could you file a draft-PR for your patch, just so we can get those tests going?

BrainSlayer commented 4 years ago

@Ornias1993 i want to get some feedback from @nivedita76 first to make sure. i compiled it locally already with no issue

BrainSlayer commented 4 years ago

and we need to ask @behlendorf to change the testsystem to include this scenario.

PrivatePuffin commented 4 years ago

@brainslayer I know, thats not why i'm asking... I was asking to make sure it doesn't crap out on the existing tests.

BrainSlayer commented 4 years ago

@Ornias1993 me and git a never ending story. gimme some minutes

PrivatePuffin commented 4 years ago

@BrainSlayer No worries, I think we are all used to that one by now :) Draft is the little dropdown button right of the big green submit button btw ;)

BrainSlayer commented 4 years ago

its about branching. my git tree has also another fix for my own system (backward compatiblity with dev versions). i already branched it now https://github.com/openzfs/zfs/pull/10775

terrelln commented 4 years ago

no we cannot. the in kernel zstd implementation is feature reduced and out of date

FYI I am in the process of updating it to use the latest upstream directly. The only feature not supported is calls to malloc() and free() (without ZSTD_customMem, e.g. ZSTD_createCCtx()), because there isn't a great default, and it needs to work pre-boot. But, providing a ZSTD_customMem will work just fine.

Let me know if you would be interested in using the zstd in the kernel once it has been upgraded. You can reach out via email at ${github-username}@fb.com, or I'll check back on this thread.

We can't rely on the version in the kernel, because it might mismatch the version that originally compressed your data. Simply put: If you don't want you data be corrupted you don't want to rely on the kernel ZSTD.

The zstd format is stable and zstd versions are both forward and backward compatible. The version in the kernel can understand data produced by the latest version, and visa-versa. Using a different version is not a problem.

PrivatePuffin commented 4 years ago

@terrelln ZSTD has already been implemented, there are no plans for using in-kernel zstd, talks about this have been concluded for months by now...

Anyhow: In essence this isn't really something about the ZSTD version, its about ZFS. ZFS does checksumming, even compatible versions of ZSTD might result in slightly different checksums and we need to account for it. ZFS also needs to support a lot of different platforms, both Linux and FreeBSD based. Having to rely on varying versions of ZSTD in the kernel overcomplicates things to the n'th degree.

The current implementation of zstd on zfs is quite solid, vetted just fine and this bug was fixed already without much issue.

While it's awesome you provide input on how we could this, ZSTD-on-ZFS has left the design stage by now and with the current design there isn't any benefid on using in-kernel zstd....

nivedita76 commented 4 years ago

On Fri, Aug 28, 2020 at 11:22:21AM -0700, Kjeld Schouten-Lebbing wrote:

@terrelln ZSTD has already been implemented, there are no plans for using in-kernel zstd, talks about this have been concluded for months by now...

Anyhow: In essence this isn't really something about the ZSTD version, its about ZFS. ZSTD does checksumming, even compatible versions of ZSTD might result in slightly different checksums and we need to account for it. ZSTD also needs to support a lot of different platforms, both Linux and FreeBSD based. Having to rely on varying versions of ZSTD in the kernel overcomplicates things to the n'th degree.

The current implementation of zstd on zfs is quite solid, vetted just fine and this bug was fixed already without much issue.

While it's awesome you provide input on how we could this, ZSTD-on-ZFS has left the design stage by now and with the current design there isn't any benefid on using in-kernel zstd....

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/openzfs/zfs/issues/10763#issuecomment-683037238

There is at least one benefit: getting rid of duplication. zstd adds about 0.5Mb to the ZFS kernel build.

Thanks.

PrivatePuffin commented 4 years ago

@nivedita76 The current ZFS version binds every ZFS version to a specific ZSTD version. Which would be a hell to undo.

You are also underestimating: It wouldn't cut 0.5mb from the kernel, we still would have to supply a ZSTD library for platforms (or kernel versions) that won't have ZSTD integrated. To make it more complicated, previously (about 3 years or so back), there where problems with zstd versions not working correctly when used with ZFS. Ohh and zfs also supports kernels without ZSTD buildin afaik.

So All things considered: We can't remove the ZSTD lib completely. We might with hard rewriting work make it an option to use a kernel zstd version, but no one is going to do the hell of work required to do so and actually keep supporting it. And for what? Because that would mean a few people (its a small minority thats doing what you are doing and a gpl violation to distribute a kernel like that) could compile it to save 0.5Mb?

All these design discussions have been gone over and over, for 3-4 years by now. There is a time and place for every discussion. Imho, this isn't it anymore. I'm not saying your opinion is irrelevant, but it isn't going to change anything.

Stability was always key for ZSTD-on-ZFS and supplying a trusted ZSTD source (which got tested into oblivion) is part of that stability guarantee.