Closed xiaoling closed 7 years ago
@xiaoling Can't think of a better solution than the random delays. If numpy already handles parallelization, you could limit DEEPDIVE_NUM_PROCESSES=1
.
Yeah, OPENBLAS_NUM_THREADS=1
seems to work without changing anything from deepdive.
This is mainly an issue with numpy/OpenBLAS that DeepDive cannot do much about. Closing since a workaround is recorded here. Let's add these stuffs to FAQ if people leave lots of reactions.
I have experienced some issue with deepdive when my udf imports numpy (or pandas).
I was able to reproduce the issue in a simple app as follows.
app.ddlog
:udf/test.py
:I got the following error message when I ran
deepdive run
:Interestingly, at first I can only reproduce this issue on one machine (40 core) but not the other machine (32 core) with an almost identical environment.
@alldefector identified that it's a concurrency issue. We narrowed it down to
compute-execute
link. Basically, if the for loop runs fast enough and two processes load numpy at the same time, it will crash. The reason that we didn't observe on the 32-core machine is probably because it didn't run the for loop fast enough. We stress-tested it withand was able to reproduce this same issue on both machines.
We looked at the core dump using gdb
and here is the stack trace:
Envs: Ubuntu 14.04 deepdive master branch numpy 1.11.1 with openblas
A work-around is to add a random time delay before each numpy import. Any suggestions? @netj