search

animetosho / ParPar

High performance PAR2 create client for NodeJS
202 stars 21 forks source link

RISCV64 on Alpine (not on Ubuntu): ../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m' #62

Open sanderjo opened 3 months ago

sanderjo commented 3 months ago

Trying to build ParPar in a docker (alpine:edge, gcc 14) on RISCV64, I get the output with errors below

As it's in rvv, I'm assuming it's ParPar specific (not something upstream)?

Is this something in the ParPar source code, or on my side?

  CC(target) Release/obj.target/gf16_rvv/gf16/gf16_shuffle128_rvv.o
In file included from ../gf16/gf16_shuffle128_rvv.c:1:
../gf16/gf16_rvv_common.h: In function 'gf16_vec_mul2_rvv':
../gf16/gf16_rvv_common.h:33:20: error: incompatible type for argument 3 of '__riscv_vxor_vx_i16m1_m'
   33 |                 v, v,
      |                    ^
      |                    |
      |                    vint16m1_t
In file included from ../gf16/gf16_rvv_common.h:9:
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: expected 'short int' but argument is of type 'vint16m1_t'
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h:13:17: error: too many arguments to function '__riscv_vxor_vx_i16m1_m'
   13 | #  define RV(f) __riscv_##f
      |                 ^~~~~~~~
../gf16/gf16_rvv_common.h:29:16: note: in expansion of macro 'RV'
   29 |         return RV(vxor_vx_i16m1_m)
      |                ^~
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: declared here
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_shuffle128_rvv.c: In function '_vlseg2e8':
../gf16/gf16_shuffle128_rvv.c:16:38: error: 'vd' undeclared (first use in this function); did you mean 'd'?
   16 |         *v0 = RV(vget_v_u8m1x2_u8m1)(vd, 0);
      |                                      ^~
      |                                      d
../gf16/gf16_shuffle128_rvv.c:16:38: note: each undeclared identifier is reported only once for each function it appears in
../gf16/gf16_shuffle128_rvv.c:15:22: warning: unused variable 'd' [-Wunused-variable]
   15 |         vuint8m1x2_t d = RV(vlseg2e8_v_u8m1x2)(src, vl);
      |                      ^
In file included from ../gf16/gf16_shuffle128_rvv.c:244:
../gf16/gf16_checksum_rvv.h: In function 'gf16_checksum_exp_rvv':
../gf16/gf16_checksum_rvv.h:71:35: error: incompatible type for argument 4 of '__riscv_vxor_vv_i16m1_m'
   71 |                         res, res, _checksum,
      |                                   ^~~~~~~~~
      |                                   |
      |                                   vint16m1_t
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: expected 'long unsigned int' but argument is of type 'vint16m1_t'
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h:13:17: error: too many arguments to function '__riscv_vxor_vv_i16m1_m'
   13 | #  define RV(f) __riscv_##f
      |                 ^~~~~~~~
../gf16/gf16_checksum_rvv.h:67:23: note: in expansion of macro 'RV'
   67 |                 res = RV(vxor_vv_i16m1_m)
      |                       ^~
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/include/riscv_vector.h:40:9: note: declared here
   40 | #pragma riscv intrinsic "vector"
      |         ^~~~~
../gf16/gf16_rvv_common.h: In function 'gf16_vec_mul2_rvv':
../gf16/gf16_rvv_common.h:37:1: warning: control reaches end of non-void function [-Wreturn-type]
   37 | }
      | ^
make: *** [gf16_rvv.target.mk:121: Release/obj.target/gf16_rvv/gf16/gf16_shuffle128_rvv.o] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git &&   cd ParPar &&    node-gyp rebuild &&     npm install' returned a non-zero code: 1
sanderjo commented 3 months ago

Dockerfile:

FROM alpine:edge

RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
    git clone https://github.com/animetosho/ParPar.git && \
    cd ParPar && \
    node-gyp rebuild && \
    npm install

Full docker build output attached

ParPar-docker-build.txt

sanderjo commented 3 months ago

FWIW: on x86, the same Dockerfile results with success, with parpar working

/ParPar/bin # fallocate -l100M blabla.bin

/ParPar/bin # node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
Input data        : 100 MiB (2000 slices from 1 file)
Recovery data     : 10 MiB (200 * 51.2 KiB slices)
Input pass(es)    : 1, processing 200 * 51.2 KiB chunks per pass
Read buffer size  : 4045.05 KiB * max 8 buffers
Hash method       : AVX512 (input), AVX512VL (recovery)

11th Gen Intel(R) Core(TM) i3-1115G4 @ 3.00GHz
  Multiply method : Affine (GFNI+AVX512) with 4480 B loop tiling, 4 threads
  Input batching  : 12 chunks, 2 batches
  Memory Usage    : 11.24 MiB (200 * 51.38 KiB chunks + 1233 KiB transfer buffer)

Finished          : 100.00%
Processing time   : 0.263 s
/ParPar/bin # ls -al
total 113860
drwxr-xr-x    1 root     root          4096 Aug 22 09:56 .
drwxr-xr-x    1 root     root          4096 Aug 22 09:28 ..
-rw-r--r--    1 root     root     104857600 Aug 22 09:54 blabla.bin
-rw-r--r--    1 root     root         40424 Aug 22 09:56 parfiles.par2
-rw-r--r--    1 root     root         92924 Aug 22 09:56 parfiles.vol000+001.par2
-rw-r--r--    1 root     root        145424 Aug 22 09:56 parfiles.vol001+002.par2
-rw-r--r--    1 root     root        290728 Aug 22 09:56 parfiles.vol003+004.par2
-rw-r--r--    1 root     root        541032 Aug 22 09:56 parfiles.vol007+008.par2
-rw-r--r--    1 root     root       1001336 Aug 22 09:56 parfiles.vol015+016.par2
-rw-r--r--    1 root     root       1881640 Aug 22 09:56 parfiles.vol031+032.par2
-rw-r--r--    1 root     root       3601944 Aug 22 09:56 parfiles.vol063+064.par2
-rw-r--r--    1 root     root       4074444 Aug 22 09:56 parfiles.vol127+073.par2
-rw-r--r--    1 root     root            26 Aug 22 09:26 parpar.cmd
-rwxr-xr-x    1 root     root         30842 Aug 22 09:26 parpar.js
/ParPar/bin #
animetosho commented 3 months ago

Thanks.

Could you try again using the dev branch instead of master?

sanderjo commented 3 months ago

build-parpar-riscv64.log Ah, much better. TIL!

But: failure later on:


ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git &&   cd ParPar &&    git checkout dev &&     node-gyp rebuild &&     npm install' returned a non-zero code: 1

``
animetosho commented 3 months ago

Thanks again.

Hmm, can you try searching for and deleting , "PARPAR_LIBDL_SUPPORT" in binding.gyp and try again?

sanderjo commented 3 months ago 
/ParPar # git checkout dev
branch 'dev' set up to track 'origin/dev'.
Switched to a new branch 'dev'
/ParPar # find . | grep binding.gyp
./binding.gyp
/ParPar # cat binding.gyp | grep PARPAR_LIBDL_SUPPORT
      "defines": ["NDEBUG", "PARPAR_LIBDL_SUPPORT"],
/ParPar # sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp
/ParPar #

OK, so let's put that into the Dockerfile

FROM alpine:edge

RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
    git clone https://github.com/animetosho/ParPar.git && \
    cd ParPar && \
    git checkout dev && \
    sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp && \
    node-gyp rebuild && \
    npm install

EDIT: A grep shows PARPAR_LIBDL_SUPPORT is not there anymore in binding.gyp, so the sed is working

... still same error:


rm -f Release/obj.target/parpar_gf_c.a Release/obj.target/parpar_gf_c.a.ar-file-list; mkdir -p `dirname Release/obj.target/parpar_gf_c.a`
ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok 
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git &&   cd ParPar &&    git checkout dev &&     sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp &&    node-gyp rebuild &&     npm install' returned a non-zero code: 1
``
animetosho commented 3 months ago

I don't really know then. Maybe some issue with RVV and LTO?

Try replacing the node-gyp rebuild command with LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild to disable building with LTO.

sanderjo commented 3 months ago

I'll try.

In the meantime I had already something (likely?) ugly (based on https://github.com/nodejs/node/commit/bec959ef8b97b5a589775e554964cb14548c9895) , but still error


 sed -i -e 's/"-march=rv64gcv"/"-march=rv64gcv",  "-fno-lto"/g' binding.gyp
 sed -i -e 's/"-march=rv64gcv_zvbc1"/"-march=rv64gcv_zvbc1",  "-fno-lto"/g' binding.gyp

/ParPar # cat binding.gyp | grep  'fno-lto'
          "cflags": ["-mfpu=fp-armv8","-fno-lto"],
          "cxxflags": ["-mfpu=fp-armv8","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=fp-armv8","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=fp-armv8","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
          "cxxflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=neon","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
          "cxxflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"],
            "OTHER_CXXFLAGS": ["-mfpu=neon","-fno-lto"]
          "cflags": ["-mfpu=neon","-fno-lto"],
            "OTHER_CFLAGS": ["-mfpu=neon","-fno-lto"]
              "cflags": ["-march=rv64gcv",  "-fno-lto"],
              "cxxflags": ["-march=rv64gcv",  "-fno-lto"],
                "OTHER_CFLAGS": ["-march=rv64gcv",  "-fno-lto"],
                "OTHER_CXXFLAGS": ["-march=rv64gcv",  "-fno-lto"],
              "cflags": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
              "cxxflags": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
                "OTHER_CFLAGS": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
                "OTHER_CXXFLAGS": ["-march=rv64gcv_zvbc1",  "-fno-lto"],
sanderjo commented 3 months ago

same error with LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild

ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
make: Leaving directory '/ParPar/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
gyp ERR! System Linux 6.1.15
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /ParPar
gyp ERR! node -v v20.15.1
gyp ERR! node-gyp -v v10.1.0
gyp ERR! not ok
npm error code 1
npm error path /ParPar
npm error command failed
npm error command sh -c node-gyp rebuild
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-08-22T12_54_27_970Z-debug-0.log
The command '/bin/sh -c git clone https://github.com/animetosho/ParPar.git &&   cd ParPar &&    git checkout dev &&     grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l &&       sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp &&        grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l  &&       LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild &&         npm install' returned a non-zero code: 1
sanderjo commented 3 months ago

build_parpar-ubuntu2410-riscv64.log Progress?

I built from Ubuntu:24.10 (also with gcc-14.2.0, still on RISCV64), and that works. See tail below.

With very clean Dockerfile (no specials):

FROM ubuntu:24.10

RUN apt update

RUN apt install -y nodejs npm python3 gcc g++ make automake autoconf git

RUN \
    git clone https://github.com/animetosho/ParPar.git && \
    cd ParPar && \
    git checkout dev && \
    node-gyp rebuild && \
    npm install

Which binary can I check for rvv commands (with objdump)? I only see .js and . json files, and there must be a binary?

root@78389c30ed8a:/ParPar/bin# fallocate -l100M blabla.bin
root@78389c30ed8a:/ParPar/bin# node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
Input data        : 100 MiB (2000 slices from 1 file)
Recovery data     : 10 MiB (200 * 51.2 KiB slices)
Input pass(es)    : 1, processing 200 * 51.2 KiB chunks per pass
Read buffer size  : 4045.05 KiB * max 8 buffers
Hash method       : Scalar+Generic (input), Scalar (recovery)

Spacemit(R) X60
  Multiply method : Shuffle-128 (RVV) with 17.13 KiB loop tiling, 8 threads
  Input batching  : 12 chunks, 2 batches
  Memory Usage    : 11.22 MiB (200 * 51.31 KiB chunks + 1231.5 KiB transfer buffer)

Finished          : 100.00%
Processing time   : 4.256 s
rm -f Release/obj.target/hasher_sse2.a Release/obj.target/hasher_sse2.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_sse2.a`
ar crs Release/obj.target/hasher_sse2.a @Release/obj.target/hasher_sse2.a.ar-file-list
  COPY Release/hasher_sse2.a
  CXX(target) Release/obj.target/hasher_sve2/hasher/hasher_sve2.o
rm -f Release/obj.target/hasher_sve2.a Release/obj.target/hasher_sve2.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_sve2.a`
ar crs Release/obj.target/hasher_sve2.a @Release/obj.target/hasher_sve2.a.ar-file-list
  COPY Release/hasher_sve2.a
  CXX(target) Release/obj.target/hasher_xop/hasher/hasher_xop.o
rm -f Release/obj.target/hasher_xop.a Release/obj.target/hasher_xop.a.ar-file-list; mkdir -p `dirname Release/obj.target/hasher_xop.a`
ar crs Release/obj.target/hasher_xop.a @Release/obj.target/hasher_xop.a.ar-file-list
  COPY Release/hasher_xop.a
  CC(target) Release/obj.target/parpar_gf_c/gf16/opencl-include/cl.o
  CC(target) Release/obj.target/parpar_gf_c/gf16/gfmat_coeff.o
rm -f Release/obj.target/parpar_gf_c.a Release/obj.target/parpar_gf_c.a.ar-file-list; mkdir -p `dirname Release/obj.target/parpar_gf_c.a`
ar crs Release/obj.target/parpar_gf_c.a @Release/obj.target/parpar_gf_c.a.ar-file-list
  COPY Release/parpar_gf_c.a
  CXX(target) Release/obj.target/parpar_gf/src/gf.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_cpu.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl.o
  CXX(target) Release/obj.target/parpar_gf/gf16/controller_ocl_init.o
  SOLINK_MODULE(target) Release/obj.target/parpar_gf.node
  COPY Release/parpar_gf.node
make: Leaving directory '/ParPar/build'
gyp info ok

added 2 packages, and audited 3 packages in 5m

found 0 vulnerabilities
 ---> Removed intermediate container 0405e39b3f7b
 ---> 946014a52d12
Successfully built 946014a52d12
animetosho commented 3 months ago

Hmm, maybe node-gyp is overriding the CFLAGS.
I really don't know, so I can only offer 'shot-in-the-dark' suggestions:
Does replacing node-gyp rebuild with node-gyp rebuild --enable_lto=false do anything?

Progress?

My guess is it's something with the musl build chain that doesn't like it. I think you previously built this fine on Ubuntu.

Which binary can I check for rvv commands (with objdump)?

The library should be build/Release/parpar_gf.node

Your output shows Multiply method : Shuffle-128 (RVV) though, so RVV is compiled in.

sanderjo commented 3 months ago

Does replacing node-gyp rebuild with node-gyp rebuild --enable_lto=false do anything?

Same error.

Shall I close this issue as "Alpine specific"?

lto1: fatal error: target specific builtin not available
compilation terminated.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/riscv64-alpine-linux-musl/14.2.0/../../../../riscv64-alpine-linux-musl/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make: *** [parpar_gf.target.mk:169: Release/obj.target/parpar_gf.node] Error 1
sanderjo commented 3 months ago

In the Ubuntu build version indeed rvv commands inside /ParPar/build/Release/parpar_gf.node

Cool

root@b5a1c081e006:~# objdump -d /ParPar/build/Release/parpar_gf.node | awk '{ print $3 }' | sort -u | grep -e ^v | head
vadd.vv
vand.vi
vand.vv
vfirst.m
vl1r.v
vl1re16.v
vle8.v
vlm.v
vlseg2e16.v
vlseg2e8.v

EDIT

I could have found it:

/ParPar# find . -type f -executable -print | grep -vi -e js$ -e sample$ -e sh$
./build/Release/parpar_gf.node
./build/Release/obj.target/parpar_gf.node

Thanks

animetosho commented 3 months ago

Shall I close this issue as "Alpine specific"?

I'm willing to diagnose this as long as you're willing to test out my random shots in the dark. I don't really have your environment set up, so it's hard for me to replicate.

If you are willing to diagnose further:
Can you try a grep lto build/* to see if there's any LTO refs in the build scripts?
Also a node-gyp rebuild -v to show all the compilation commands might yield some more info.

It's odd that it looks like LTO is being invoked despite attempts to disable it.

sanderjo commented 3 months ago

I'm willing to diagnose this as long as you're willing to test out my random shots in the dark.

Thank you.

I don't really have your environment set up, so it's hard for me to replicate.

Still my offer to you: via IPv6 access to my riscv system

I did stuff, and I'm dumping it here, without thinking about it too much build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log

TLDR: it now builds (maybe because of ignoring the returncode in the Dockerfile). EDIT: After a manual npm install async I can run parpar.

Docker build command longer, but nothing special:

sander@k1:~/parpar-docker$ docker build -t no-lto---verbose-build . -f Dockerfile.no-lto---verbose-build 2>&1 | tee build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log

Dockerfile now with all precautions, plus ignoring the output of the build 

$ cat Dockerfile.no-lto---verbose-build
FROM alpine:edge

RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
    git clone https://github.com/animetosho/ParPar.git && \
    cd ParPar && \
    git checkout dev && \
    grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l && \
    sed -i -e 's/, "PARPAR_LIBDL_SUPPORT"//g' binding.gyp && \
    grep PARPAR_LIBDL_SUPPORT binding.gyp | wc -l  && \
    LDFLAGS=-fno-lto CFLAGS=-fno-lto CXXFLAGS=-fno-lto node-gyp rebuild -v ; exit 0 \
    npm install
[build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log](https://github.com/user-attachments/files/16739535/build-parpar-riscv64---2024-08-25---no-lto---verbose-build.log)

EDIT: the below solved with npm install async

/ParPar/bin # fallocate -l100M blabla.bin
/ParPar/bin # 
/ParPar/bin # node parpar.js  -s2000 --min-input-slices=1 -r10% -d pow2 -o parfiles *bin
node:internal/modules/cjs/loader:1148
  throw err;
  ^

Error: Cannot find module 'async'
Require stack:
- /ParPar/lib/par2.js
- /ParPar/lib/parpar.js
- /ParPar/bin/parpar.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/ParPar/lib/par2.js:5:13)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/ParPar/lib/par2.js',
    '/ParPar/lib/parpar.js',
    '/ParPar/bin/parpar.js'
  ]
}

Node.js v20.15.1

  g++ -o Release/obj.target/parpar_gf.node -shared -pthread -rdynamic -flto=4 -fuse-linker-plugin -ffat-lto-objects  -fno-lto -Wl,-soname=parpar_gf.node -Wl,--start-group Release/obj.target/parpar_gf/src/gf.o Release/obj.target/parpar_gf/gf16/controller.o Release/obj.target/parpar_gf/gf16/controller_cpu.o Release/obj.target/parpar_gf/gf16/controller_ocl.o Release/obj.target/parpar_gf/gf16/controller_ocl_init.o Release/obj.target/parpar_gf_c.a Release/obj.target/gf16.a Release/obj.target/gf16_generic.a Release/obj.target/gf16_sse2.a Release/obj.target/gf16_ssse3.a Release/obj.target/gf16_avx.a Release/obj.target/gf16_avx2.a Release/obj.target/gf16_avx512.a Release/obj.target/gf16_vbmi.a Release/obj.target/gf16_gfni.a Release/obj.target/gf16_gfni_avx2.a Release/obj.target/gf16_gfni_avx512.a Release/obj.target/gf16_gfni_avx10.a Release/obj.target/gf16_neon.a Release/obj.target/gf16_sha3.a Release/obj.target/gf16_sve.a Release/obj.target/gf16_sve2.a Release/obj.target/gf16_rvv.a Release/obj.target/gf16_rvv_zvbc.a Release/obj.target/hasher.a Release/obj.target/hasher_sse2.a Release/obj.target/hasher_clmul.a Release/obj.target/hasher_xop.a Release/obj.target/hasher_bmi1.a Release/obj.target/hasher_avx2.a Release/obj.target/hasher_avx512.a Release/obj.target/hasher_avx512vl.a Release/obj.target/hasher_armcrc.a Release/obj.target/hasher_neon.a Release/obj.target/hasher_neoncrc.a Release/obj.target/hasher_sve2.a Release/obj.target/hasher_rvzbc.a Release/obj.target/hasher_c.a -Wl,--end-group 
  ln -f "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node" 2>/dev/null || (rm -rf "Release/parpar_gf.node" && cp -af "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node")
make: Leaving directory '/ParPar/build'
gyp info ok 
 ---> Removed intermediate container 8baf2d0c2ba8
 ---> 56f80b5dc1b7
Successfully built 56f80b5dc1b7
Successfully tagged no-lto---verbose-build:latest
sander@k1:~/parpar-docker$

Binaries are there

/ParPar # find . -type f -executable -print | grep -vi -e js$ -e sample$ -e sh$
./build/Release/parpar_gf.node
./build/Release/obj.target/parpar_gf.node

/ParPar # grep lto build/*
build/config.gypi:    "enable_lto": "true",
build/gf16.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx2.target.mk:   -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx2.target.mk:   -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx512.target.mk:     -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_avx512.target.mk:     -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_generic.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_generic.target.mk:    -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni.target.mk:   -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni.target.mk:   -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx10.target.mk:     -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx10.target.mk:     -flto=4 -fuse-linker-plugin -ffat-lto-objects 
build/gf16_gfni_avx2.target.mk:  -flto=4 -fuse-linker-plugin -ffat-lto-objects
animetosho commented 3 months ago

Thanks, looks like LTO is still being enabled.

Can you try with this Dockerfile instead?

FROM alpine:edge

RUN apk update
RUN apk add nodejs npm python3 gcc g++ make automake autoconf git
RUN \
    git clone https://github.com/animetosho/ParPar.git && \
    cd ParPar && \
    git checkout dev && \
    node-gyp rebuild --enable_lto=false -v && \
    grep lto build/* ; exit 0 \
    npm install
sanderjo commented 3 months ago

That one built without problems.

build.Dockerfile.animetosho_2024-08-26.log

Still strange: an error when trying to run parpar.js ... which is solved by manuall installing async. Which I tried putting into Dockerfile, without solving it.

sander@k1:~/parpar-docker$ docker run -it parpar-animetosho_2024-08-26 sh
/ # cd ParPar/bin

/ParPar/bin # node parpar.js --help 
node:internal/modules/cjs/loader:1148
  throw err;
  ^

Error: Cannot find module 'async'
Require stack:
- /ParPar/lib/par2.js
- /ParPar/lib/parpar.js
- /ParPar/bin/parpar.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/ParPar/lib/par2.js:5:13)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/ParPar/lib/par2.js',
    '/ParPar/lib/parpar.js',
    '/ParPar/bin/parpar.js'
  ]
}

Node.js v20.15.1

/ParPar/bin # npm install async

added 2 packages, and audited 3 packages in 9s

found 0 vulnerabilities
/ParPar/bin #

/ParPar/bin # node parpar.js --help 2>&1 | head -5
ParPar v0.4.4-dev
High performance PAR2 creation tool
-----------------------------------

Usage: parpar -s <slice_size/count> -o <output> [options] [--] <input1> [<input2>...]
/ParPar/bin #

Next: building without the hack exit 0, and with the &&

The grep output:

build/config.gypi:    "enable_lto": "false",
  g++ -o Release/obj.target/parpar_gf.node -shared -pthread -rdynamic  -Wl,-soname=parpar_gf.node -Wl,--start-group Release/obj.target/parpar_gf/src/gf.o Release/obj.target/parpar_gf/gf16/controller.o Release/obj.target/parpar_gf/gf16/controller_cpu.o Release/obj.target/parpar_gf/gf16/controller_ocl.o Release/obj.target/parpar_gf/gf16/controller_ocl_init.o Release/obj.target/parpar_gf_c.a Release/obj.target/gf16.a Release/obj.target/gf16_generic.a Release/obj.target/gf16_sse2.a Release/obj.target/gf16_ssse3.a Release/obj.target/gf16_avx.a Release/obj.target/gf16_avx2.a Release/obj.target/gf16_avx512.a Release/obj.target/gf16_vbmi.a Release/obj.target/gf16_gfni.a Release/obj.target/gf16_gfni_avx2.a Release/obj.target/gf16_gfni_avx512.a Release/obj.target/gf16_gfni_avx10.a Release/obj.target/gf16_neon.a Release/obj.target/gf16_sha3.a Release/obj.target/gf16_sve.a Release/obj.target/gf16_sve2.a Release/obj.target/gf16_rvv.a Release/obj.target/gf16_rvv_zvbc.a Release/obj.target/hasher.a Release/obj.target/hasher_sse2.a Release/obj.target/hasher_clmul.a Release/obj.target/hasher_xop.a Release/obj.target/hasher_bmi1.a Release/obj.target/hasher_avx2.a Release/obj.target/hasher_avx512.a Release/obj.target/hasher_avx512vl.a Release/obj.target/hasher_armcrc.a Release/obj.target/hasher_neon.a Release/obj.target/hasher_neoncrc.a Release/obj.target/hasher_sve2.a Release/obj.target/hasher_rvzbc.a Release/obj.target/hasher_c.a -Wl,--end-group 
  ln -f "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node" 2>/dev/null || (rm -rf "Release/parpar_gf.node" && cp -af "Release/obj.target/parpar_gf.node" "Release/parpar_gf.node")
make: Leaving directory '/ParPar/build'
gyp info ok 
build/config.gypi:    "enable_lto": "false",
 ---> Removed intermediate container e7011cc871f2
 ---> 4076e6b9233a
Successfully built 4076e6b9233a
Successfully tagged parpar-animetosho_2024-08-26:latest

real    5m25.475s
user    0m0.512s
sys 0m0.574s
animetosho commented 3 months ago

That one built without problems.

Yay!

Still strange: an error when trying to run parpar.js ... which is solved by manuall installing async

Usually the npm install will install the dependency.