search

yjxiong / temporal-segment-networks

Code & Models for Temporal Segment Networks (TSN) in ECCV 2016
BSD 2-Clause "Simplified" License
1.53k stars 477 forks source link

exited on signal 7 (Bus error). #265

Open Hiker01 opened 5 years ago

Hiker01 commented 5 years ago

I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).

I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934 I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss) I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001 I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332 I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305 Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date PC: @ 0x7f1f2c367f56 (unknown) PC: @ 0x7fc173393f56 (unknown) PC: @ 0x7fef15a8df56 (unknown) PC: @ 0x7ff6e251ef56 (unknown)

mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).