search

opencv / opencv

Open Source Computer Vision Library
https://opencv.org
Apache License 2.0
78.6k stars 55.78k forks source link

shared:ERROR: If pthreads and memory growth are enabled, WASM_MEM_MAX must be set #14691

Open DieFishEye opened 5 years ago

DieFishEye commented 5 years ago

/home/z/emsdk/emscripten/incoming/em++ -s WASM=1 -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Winconsistent-missing-override -Wno-delete-non-virtual-dtor -Wno-unnamed-type-template-args -Wno-comment -fdiagnostics-show-option -pthread -Qunused-arguments -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -DNDEBUG -O2 -DNDEBUG -Wl,--gc-sections -O2 --memory-init-file 0 -s TOTAL_MEMORY=134217728 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s SINGLE_FILE=1 -s EXPORT_NAME="'cv'" -s DEMANGLE_SUPPORT=1 -s FORCE_FILESYSTEM=1 --use-preload-plugins --bind --post-js /home/z/opencv/modules/js/src/helpers.js -Wno-missing-prototypes @CMakeFiles/opencv_js.dir/objects1.rsp -o ../../bin/opencv_js.js @CMakeFiles/opencv_js.dir/linklibs.rsp root:WARNING: USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 shared:ERROR: If pthreads and memory growth are enabled, WASM_MEM_MAX must be set modules/js/CMakeFiles/opencv_js.dir/build.make:179: recipe for target 'bin/opencv_js.js' failed make[3]: *** [bin/opencv_js.js] Error 1 make[3]: Leaving directory '/home/z/opencv/build_wasm' CMakeFiles/Makefile2:1514: recipe for target 'modules/js/CMakeFiles/opencv_js.dir/all' failed make[2]: *** [modules/js/CMakeFiles/opencv_js.dir/all] Error 2 make[2]: Leaving directory '/home/z/opencv/build_wasm' CMakeFiles/Makefile2:1479: recipe for target 'modules/js/CMakeFiles/opencv.js.dir/rule' failed make[1]: *** [modules/js/CMakeFiles/opencv.js.dir/rule] Error 2 make[1]: Leaving directory '/home/z/opencv/build_wasm' Makefile:390: recipe for target 'opencv.js' failed make: *** [opencv.js] Error 2 Traceback (most recent call last): File "./platforms/js/build_js.py", line 235, in <module> builder.build_opencvjs() File "./platforms/js/build_js.py", line 170, in build_opencvjs execute(["make", "-j", str(multiprocessing.cpu_count()), "opencv.js"]) File "./platforms/js/build_js.py", line 23, in execute raise Fail("Child returned: %s" % retcode) __main__.Fail: Child returned: 2 z@z:~/opencv$

dkurt commented 5 years ago

Any chance to know how to reproduce it?

DieFishEye commented 5 years ago

build opencv js failure

alalek commented 5 years ago

@DieFishEye Please compare your logs with public CI logs and provide necessary information about changes.

DieFishEye commented 5 years ago

@DieFishEye Please compare your logs with public CI logs and provide necessary information about changes.

The system I used was compiled by Ubuntu, according to the compilation process on the official website

Wenzhao-Xiang commented 5 years ago

I got the same error when compiling my own project using pthread and memory growth at the same time. After setting -s WASM_MEM_MAX=${max wasm memory size}, the error disappeared and the compiling passed. However, it's about 5-6x slower than the version without memory growth.

JsBlueCat commented 5 years ago

/home/z/emsdk/emscripten/incoming/em++ -s WASM=1 -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Winconsistent-missing-override -Wno-delete-non-virtual-dtor -Wno-unnamed-type-template-args -Wno-comment -fdiagnostics-show-option -pthread -Qunused-arguments -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -DNDEBUG -O2 -DNDEBUG -Wl,--gc-sections -O2 --memory-init-file 0 -s TOTAL_MEMORY=134217728 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s SINGLE_FILE=1 -s EXPORT_NAME="'cv'" -s DEMANGLE_SUPPORT=1 -s FORCE_FILESYSTEM=1 --use-preload-plugins --bind --post-js /home/z/opencv/modules/js/src/helpers.js -Wno-missing-prototypes @CMakeFiles/opencv_js.dir/objects1.rsp -o ../../bin/opencv_js.js @CMakeFiles/opencv_js.dir/linklibs.rsp root:WARNING: USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 shared:ERROR: If pthreads and memory growth are enabled, WASM_MEM_MAX must be set modules/js/CMakeFiles/opencv_js.dir/build.make:179: recipe for target 'bin/opencv_js.js' failed make[3]: *** [bin/opencv_js.js] Error 1 make[3]: Leaving directory '/home/z/opencv/build_wasm' CMakeFiles/Makefile2:1514: recipe for target 'modules/js/CMakeFiles/opencv_js.dir/all' failed make[2]: *** [modules/js/CMakeFiles/opencv_js.dir/all] Error 2 make[2]: Leaving directory '/home/z/opencv/build_wasm' CMakeFiles/Makefile2:1479: recipe for target 'modules/js/CMakeFiles/opencv.js.dir/rule' failed make[1]: *** [modules/js/CMakeFiles/opencv.js.dir/rule] Error 2 make[1]: Leaving directory '/home/z/opencv/build_wasm' Makefile:390: recipe for target 'opencv.js' failed make: *** [opencv.js] Error 2 Traceback (most recent call last): File "./platforms/js/build_js.py", line 235, in <module> builder.build_opencvjs() File "./platforms/js/build_js.py", line 170, in build_opencvjs execute(["make", "-j", str(multiprocessing.cpu_count()), "opencv.js"]) File "./platforms/js/build_js.py", line 23, in execute raise Fail("Child returned: %s" % retcode) __main__.Fail: Child returned: 2 z@z:~/opencv$

i look into build_js.py file,and find 

def get_build_flags(self):
        flags = ""
        if self.options.build_wasm:
            flags += "-s WASM=1 "
        elif self.options.disable_wasm:
            flags += "-s WASM=0 "
        if self.options.enable_exception:
            flags += "-s DISABLE_EXCEPTION_CATCHING=0 "
        return flags

so , i add --disable_wasm --enable_exception

JsBlueCat commented 5 years ago

I got the same error when compiling my own project using pthread and memory growth at the same time. After setting -s WASM_MEM_MAX=${max wasm memory size}, the error disappeared and the compiling passed. However, it's about 5-6x slower than the version without memory growth.

i use your method, but it throws out this question

emcc:WARNING: ignoring unsupported linker flag: `--gc-sections`
shared:ERROR: invalid forced library: libc++abi
Santhosh-KS commented 5 years ago

@JsBlueCat

I'm also stuck with the same error. Did you find any solution for the same?

emcc:WARNING: ignoring unsupported linker flag: `--gc-sections`
shared:ERROR: invalid forced library: libc++abi
littleylv commented 5 years ago

@JsBlueCat @Santhosh-KS Have you solved that problem?

OS: macOS 10.14.5

Santhosh-KS commented 5 years ago

@littleylv unfortunately still stuck with the same error. Just an fyi I'm trying to build it on Ubuntu 18

lawsonfulton commented 5 years ago

I've got the same problem on an unrelated project. I fixed it by removing -s DEMANGLE_SUPPORT=1

Santhosh-KS commented 5 years ago

Hi @lawsonfulton,

Thanks for the inputs. Unfortunately, using that flag is causing lot of other errors.

[  1%] Building CXX object 3rdparty/protobuf/CMakeFiles/libprotobuf.dir/src/google/protobuf/arena.cc.o
cd /home/test/opencv/sant_wasm/3rdparty/protobuf && /home/santhosh/work/emsdk/emscripten/incoming/em++   @CMakeFiles/libprotobuf.dir/includes_CXX.rsp -s WASM=1 -s WASM_MEM_MAX=100MB -s DEMANGLE_SUPPORT=1   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wstrict-prototypes -Winit-self -Wpointer-arith -Wuninitialized -Winit-self -Wno-delete-non-virtual-dtor -Wno-unnamed-type-template-args -Wno-comment -fdiagnostics-show-option -pthread -Qunused-arguments -ffunction-sections -fdata-sections  -fvisibility=hidden -fvisibility-inlines-hidden -DNDEBUG -O2  -DNDEBUG -fPIC   -std=c++11 -o CMakeFiles/libprotobuf.dir/src/google/protobuf/arena.cc.o -c /home/test/opencv/3rdparty/protobuf/src/google/protobuf/arena.cc
/home/test/opencv/3rdparty/zlib/gzlib.c:252:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        LSEEK(state->fd, 0, SEEK_END);  /* so gzoffset() is correct */
        ^
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/home/test/opencv/3rdparty/zlib/gzlib.c:252:9: note: did you mean 'fseek'?
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/home/santhosh/work/emsdk/emscripten/incoming/system/include/libc/stdio.h:75:5: note: 'fseek' declared here
int fseek(FILE *, long, int);
    ^
/home/test/opencv/3rdparty/zlib/gzlib.c:258:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        state->start = LSEEK(state->fd, 0, SEEK_CUR);
                       ^
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/home/test/opencv/3rdparty/zlib/gzlib.c:359:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
        ^
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/home/test/opencv/3rdparty/zlib/gzlib.c:400:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
              ^
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/home/test/opencv/3rdparty/zlib/gzlib.c:496:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    offset = LSEEK(state->fd, 0, SEEK_CUR);
             ^
/home/test/opencv/3rdparty/zlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
5 errors generated.
shared:ERROR: compiler frontend failed to generate LLVM bitcode, halting
/home/test/opencv/3rdparty/zlib/gzread.c:35:15: error: implicit declaration of function 'read' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = read(state->fd, buf + *have, get);
littleylv commented 5 years ago

@Santhosh-KS @lawsonfulton I change the flag here:

    def get_build_flags(self):
        flags = "-s WASM_MEM_MAX=512MB -s DEMANGLE_SUPPORT=0 "  # SEE HERE!!!!!!!
        if self.options.build_wasm:
            flags += "-s WASM=1 "
        elif self.options.disable_wasm:
            flags += "-s WASM=0 "
        if self.options.enable_exception:
            flags += "-s DISABLE_EXCEPTION_CATCHING=0 "
        return flags

But also get the error: shared:ERROR: invalid forced library: libc++abi

JsBlueCat commented 5 years ago

@JsBlueCat @Santhosh-KS Have you solved that problem?

OS: macOS 10.14.5

I have solved the problem,it may be caused by your emscript version. Down your version and it should work.

may be this help

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Clone, pull, and build the latest "sdk" Emscripten environment
./emsdk install sdk-tag-1.38.31-64bit

# Set the "incoming SDK" as the active version.
./emsdk activate sdk-tag-1.38.31-64bit

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

the last sdk may have some errors , choose the 1.38

Santhosh-KS commented 5 years ago

@JsBlueCat

Thanks for the input. Yes I was able to build the opencv.js with Your suggested step to fall back to the older version of emsdk.

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Clone, pull, and build the latest "sdk" Emscripten environment
./emsdk install sdk-tag-1.38.31-64bit

# Set the "incoming SDK" as the active version.
./emsdk activate sdk-tag-1.38.31-64bit

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

However I'm not able to build wasm file with the following command 

python ./platforms/js/build_js.py my_fix_js --build_wasm

And I'm getting following error. It is a different error compared to other one. Notiece the --build_wasm option in the above command.

Fatal: error in validating output
shared:ERROR: '/home/test/emsdk/binaryen/tag-1.38.31_64bit_binaryen/bin/asm2wasm ../../bin/opencv_js.temp.asm.js --total-memory=134217728 --trap-mode=allow -O2 --mem-init=../../bin/opencv_js.js.mem --mem-base=1024 --mem-max=104857600 --wasm-only -o ../../bin/opencv_js.wasm --mvp-features' failed (1)
modules/js/CMakeFiles/opencv_js.dir/build.make:178: recipe for target 'bin/opencv_js.js' failed
slushnys commented 5 years ago

@JsBlueCat

Thanks for the input. Yes I was able to build the opencv.js with Your suggested step to fall back to the older version of emsdk.

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Clone, pull, and build the latest "sdk" Emscripten environment
./emsdk install sdk-tag-1.38.31-64bit

# Set the "incoming SDK" as the active version.
./emsdk activate sdk-tag-1.38.31-64bit

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

However I'm not able to build wasm file with the following command

python ./platforms/js/build_js.py my_fix_js --build_wasm

And I'm getting following error. It is a different error compared to other one. Notiece the --build_wasm option in the above command.

Fatal: error in validating output
shared:ERROR: '/home/test/emsdk/binaryen/tag-1.38.31_64bit_binaryen/bin/asm2wasm ../../bin/opencv_js.temp.asm.js --total-memory=134217728 --trap-mode=allow -O2 --mem-init=../../bin/opencv_js.js.mem --mem-base=1024 --mem-max=104857600 --wasm-only -o ../../bin/opencv_js.wasm --mvp-features' failed (1)
modules/js/CMakeFiles/opencv_js.dir/build.make:178: recipe for target 'bin/opencv_js.js' failed

Even without the same errors you have and successful build of opencv.js file, I still can't find .wasm file.

There's a project based on wasm of openCV if you're interested. I still haven't tried it out myself unfortunatelly.

https://hacks.mozilla.org/2017/09/bootcamps-webassembly-and-computer-vision/ https://github.com/Web-Sight/WebSight

LanceXJH commented 5 years ago

@JsBlueCat

Thanks for the input. Yes I was able to build the opencv.js with Your suggested step to fall back to the older version of emsdk.

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Clone, pull, and build the latest "sdk" Emscripten environment
./emsdk install sdk-tag-1.38.31-64bit

# Set the "incoming SDK" as the active version.
./emsdk activate sdk-tag-1.38.31-64bit

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

However I'm not able to build wasm file with the following command

python ./platforms/js/build_js.py my_fix_js --build_wasm

And I'm getting following error. It is a different error compared to other one. Notiece the --build_wasm option in the above command.

Fatal: error in validating output
shared:ERROR: '/home/test/emsdk/binaryen/tag-1.38.31_64bit_binaryen/bin/asm2wasm ../../bin/opencv_js.temp.asm.js --total-memory=134217728 --trap-mode=allow -O2 --mem-init=../../bin/opencv_js.js.mem --mem-base=1024 --mem-max=104857600 --wasm-only -o ../../bin/opencv_js.wasm --mvp-features' failed (1)
modules/js/CMakeFiles/opencv_js.dir/build.make:178: recipe for target 'bin/opencv_js.js' failed

It works for me with --build_wasm command. You might need to include the path of emsdk as well, for example: python ./platforms/js/build_js.py --emscripten_dir ~/tutorial/emsdk/emscripten/tag-1.38.31 build_wasm --build_wasm

dkurt commented 5 years ago

@Wenzhao-Xiang, Can you please also experiment with flags += "-s USE_PTHREADS=0 " at get_build_flags in build_js.py?

    def get_build_flags(self):
        flags = ""
        flags += "-s USE_PTHREADS=0 "
        if self.options.build_wasm:
            flags += "-s WASM=1 "
        elif self.options.disable_wasm:
            flags += "-s WASM=0 "
        if self.options.enable_exception:
            flags += "-s DISABLE_EXCEPTION_CATCHING=0 "
        return flags

Using just this flag helped me to build opencv.js with the latest emscripten. It's interesting if performance is the same or not.

Wenzhao-Xiang commented 5 years ago

@dkurt Thanks for the input! I think the performance will be affected only when the pthread and allow_memory_growth are both enabled. With my investigation, the latest version of emscripten seems to enable pthread by default, which is also the root cause of this issue. Anyway, I will test the performance with my benchmark. I'm now working with opencv.js performance optimization. I will also test the performance if both pthread and allow_memory_growth are enabled.

Wenzhao-Xiang commented 5 years ago

@dkurt Here is the data:

Opencv version: 4.1.0-dev Emscripten version: latest

Test environment: System: Ubuntu 16.04 CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz Chrome version: Version 75.0.3770.90 (Official Build) unknown (64-bit)

Kernel \ Build Type Type1 Type2
GaussianBlur 3.356ms  3.334ms
cvtColor 0.518ms 0.513ms
threshold 0.202ms  0.211ms 
integral2 0.769ms 0.743ms

Type 1: -s ALLOW_MEMORY_GROWTH=1 + -s DEMANGLE_SUPPORT=0 (+ -s USE_PTHREADS=1 ) (latest version of emscripten enable pthreads building option by default) Type 2: -s ALLOW_MEMORY_GROWTH=1 + -s DEMANGLE_SUPPORT=0 + -s USE_PTHREADS=0

According to the data, seems there is no performance issue when only testing kernels if I enable pthread+allow_memory_growth. But I'm not sure whether there are performance issues for real case, because I still get the warning "USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271".

Wenzhao-Xiang commented 5 years ago

@dkurt Test with face recognition sample. Here is the result: Type 1: image Type 2: image

Seems no performance issue, too.

slushnys commented 5 years ago

Off topic: Can we talk about the memory leaks in JavaScript for example? I found out that if I create cv.Mat() instances and store them in array, even tho I delete them afterwards, its imposible to free up the memory for those allocated instances afterwards. Has anyone else had to deal with this?

dkurt commented 5 years ago

@slushnys, If you have a reproducer feel free to open a new issue. Thanks!

kavikode commented 5 years ago

I have the following: root:WARNING: USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 shared:ERROR: If pthreads and memory growth are enabled, WASM_MEM_MAX must be set

how can I fix this error?

cancerberoSgx commented 5 years ago

This error happens only with latest emscripten versions. A working version of emscripte 1.38.32. So it wasn't just an injection from https://github.com/huningxin/opencv/pull/264

I'm changing the docker section of setup tutorial so it uses a compatible emscripten version and not latest as currently does. But would be perfect just to fix this issue so we can keep up with latests emcc.

@kavikode This is working using docker : 

docker run --rm --workdir /code -v "$PWD":/code "trzeci/emscripten:sdk-tag-1.38.32-64bit" python ./platforms/js/build_js.py build