I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).
I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934
I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss)
I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001
I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332
I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305
Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date
PC: @ 0x7f1f2c367f56 (unknown)
PC: @ 0x7fc173393f56 (unknown)
PC: @ 0x7fef15a8df56 (unknown)
PC: @ 0x7ff6e251ef56 (unknown)
mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).
I rebuild caffe in docker with mpi support. However when I train with optical flow modality, I got the error :exited on signal 7 (Bus error).
I0404 08:28:17.483947 23 solver.cpp:240] Iteration 280, loss = 0.837934 I0404 08:28:17.484012 23 solver.cpp:255] Train net output #0: loss = 0.891086 (* 1 = 0.891086 loss) I0404 08:28:17.484028 23 solver.cpp:640] Iteration 280, lr = 0.001 I0404 08:28:17.499704 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 80.5912 > 40) by scale factor 0.496332 I0404 08:28:25.655087 23 solver.cpp:625] Gradient clipping: scaling down gradients (L2 norm 69.5283 > 40) by scale factor 0.575305 Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date Aborted at 1554366514 (unix time) try "date -d @1554366514" if you are using GNU date PC: @ 0x7f1f2c367f56 (unknown) PC: @ 0x7fc173393f56 (unknown) PC: @ 0x7fef15a8df56 (unknown) PC: @ 0x7ff6e251ef56 (unknown)
mpirun noticed that process rank 1 with PID 24 on node b7a4deb699f1 exited on signal 7 (Bus error).