Reproduce error: Non-existant physical address

[jieWANGforwork]

Hi!

I met this error when reproducing VQA task, could you please have a loook and give me some suggestion based on your experience? Thanks a lot!

0%| | 0/6000 [00:00<?, ?it/s][1,0]:08/15/2021 10:08:00 - INFO - main - Running training with 4 GPUs [1,0]:08/15/2021 10:08:00 - INFO - main - Num examples = 471128 [1,0]:08/15/2021 10:08:00 - INFO - main - Batch size = 1024 [1,0]:08/15/2021 10:08:00 - INFO - main - Accumulate steps = 5 [1,0]:08/15/2021 10:08:00 - INFO - main - Num steps = 6000 [1,0]:[1a62e574072d:00334] Process received signal [1,0]:[1a62e574072d:00334] Signal: Bus error (7) [1,0]:[1a62e574072d:00334] Signal code: Non-existant physical address (2) [1,0]:[1a62e574072d:00334] Failing at address: 0x7f246888f00a [1,0]:[1a62e574072d:00334] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f25b870a390] [1,0]:[1a62e574072d:00334] [ 1] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x128e0)[0x7f25aa5988e0] [1,0]:[1a62e574072d:00334] [ 2] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x12b74)[0x7f25aa598b74] [1,0]:[1a62e574072d:00334] [ 3] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x14ba5)[0x7f25aa59aba5] [1,0]:[1a62e574072d:00334] [ 4] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(mdb_get+0xbc)[0x7f25aa59b40c] [1,0]:[1a62e574072d:00334] [ 5] [1,0]:/opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x9d9d)[0x7f25aa58fd9d] [1,0]:[1a62e574072d:00334] [ 6] python(_PyCFunction_FastCallDict+0x154)[0x55f2e08c1744] [1,0]:[1a62e574072d:00334] [ 7] [1,0]:python(+0x19842c)[0x55f2e094842c] [1,0]:[1a62e574072d:00334] [ 8] python(_PyEval_EvalFrameDefault+0x30a)[0x55f2e096d38a] [1,0]:[1a62e574072d:00334] [ 9] [1,0]:python(_PyFunction_FastCallDict+0x11b)[0x55f2e0942bab] [1,0]:[1a62e574072d:00334] [10] python(_PyObject_FastCallDict+0x26f)[0x55f2e08c1b0f] [1,0]:[1a62e574072d:00334] [11] [1,0]:python(_PyObject_Call_Prepend+0x63)[0x55f2e08c66a3] [1,0]:[1a62e574072d:00334] [12] python(PyObject_Call+0x3e)[0x55f2e08c154e] [1,0]:[1a62e574072d:00334] [13] [1,0]:python(+0x16b50a)[0x55f2e091b50a] [1,0]:[1a62e574072d:00334] [14] python(_PyEval_EvalFrameDefault+0x877)[0x55f2e096d8f7] [1,0]:[1a62e574072d:00334] [15] [1,0]:python(_PyFunction_FastCallDict+0x11b)[0x55f2e0942bab] [1,0]:[1a62e574072d:00334] [16] python(_PyObject_FastCallDict+0x26f)[0x55f2e08c1b0f] [1,0]:[1a62e574072d:00334] [17] python(_PyObject_Call_Prepend+0x63)[0x55f2e08c66a3] [1,0]:[1a62e574072d:00334] [18] [1,0]:python(PyObject_Call+0x3e)[0x55f2e08c154e] [1,0]:[1a62e574072d:00334] [19] python(+0x16b50a)[0x55f2e091b50a] [1,0]:[1a62e574072d:00334] [1,0]:[20] python(_PyEval_EvalFrameDefault+0x877)[0x55f2e096d8f7] [1,0]:[1a62e574072d:00334] [21] [1,0]:python(+0x19253b)[0x55f2e094253b] [1,0]:[1a62e574072d:00334] [22] python(+0x198505)[0x55f2e0948505] [1,0]:[1a62e574072d:00334] [23] [1,0]:python(_PyEval_EvalFrameDefault+0x30a)[0x55f2e096d38a] [1,0]:[1a62e574072d:00334] [24] python(+0x191a76)[0x55f2e0941a76] [1,0]:[1a62e574072d:00334] [25] python(_PyFunction_FastCallDict+0x1bc)[0x55f2e0942c4c] [1,0]:[1a62e574072d:00334] [26] [1,0]:python(_PyObject_FastCallDict+0x26f)[0x55f2e08c1b0f] [1,0]:[1a62e574072d:00334] [27] python(_PyObject_Call_Prepend+0x63)[0x55f2e08c66a3] [1,0]:[1a62e574072d:00334] [28] [1,0]:python(PyObject_Call+0x3e)[0x55f2e08c154e] [1,0]:[1a62e574072d:00334] [29] python(+0x16b50a)[0x55f2e091b50a] [1,0]:[1a62e574072d:00334] End of error message [1,2]:[1a62e574072d:00336] Process received signal [1,2]:[1a62e574072d:00336] Signal: Bus error (7) [1,2]:[1a62e574072d:00336] Signal code: Non-existant physical address (2) [1,2]:[1a62e574072d:00336] Failing at address: 0x7f56e824f00a [1,3]:[1a62e574072d:00337] Process received signal [1,3]:[1a62e574072d:00337] Signal: Bus error (7) [1,3]:[1a62e574072d:00337] Signal code: Non-existant physical address (2) [1,3]:[1a62e574072d:00337] Failing at address: 0x7fd82888f00a [1,3]:[1a62e574072d:00337] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd96b06d390] [1,3]:[1a62e574072d:00337] [ 1] [1,3]:/opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x128e0)[0x7fd955ee38e0] [1,3]:[1a62e574072d:00337] [ 2] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x12b74)[0x7fd955ee3b74] [1,3]:[1a62e574072d:00337] [ 3] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x14ba5)[0x7fd955ee5ba5] [1,3]:[1a62e574072d:00337] [ 4] [1,3]:/opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(mdb_get+0xbc)[0x7fd955ee640c] [1,3]:[1a62e574072d:00337] [ 5] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x9d9d)[0x7fd955edad9d] [1,3]:[1a62e574072d:00337] [ 6] [1,2]:[1a62e574072d:00336] [ 0] [1,3]:python(_PyCFunction_FastCallDict+0x154)[0x5600587e3744] [1,3]:[1a62e574072d:00337] [ 7] python(+0x19842c)[0x56005886a42c] [1,3]:[1a62e574072d:00337] [ 8] [1,2]:/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f582a139390] [1,2]:[1a62e574072d:00336] [ 1] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x128e0)[0x7f58106a88e0] [1,2]:[1a62e574072d:00336] [ 2] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x12b74)[0x7f58106a8b74] [1,2]:[1a62e574072d:00336] [ 3] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x14ba5)[0x7f58106aaba5] [1,2]:[1a62e574072d:00336] [ 4] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(mdb_get+0xbc)[0x7f58106ab40c] [1,2]:[1a62e574072d:00336] [ 5] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x9d9d)[0x7f581069fd9d] [1,2]:[1a62e574072d:00336] [ 6] [1,3]:python(_PyEval_EvalFrameDefault+0x30a)[0x56005888f38a] [1,3]:[1a62e574072d:00337] [ 9] python(_PyFunction_FastCallDict+0x11b)[0x560058864bab] [1,3]:[1a62e574072d:00337] [10] [1,2]:python(_PyCFunction_FastCallDict+0x154)[0x558e13464744] [1,2]:[1a62e574072d:00336] [ 7] python(+0x19842c)[0x558e134eb42c] [1,2]:[1a62e574072d:00336] [ 8] [1,3]:python(_PyObject_FastCallDict+0x26f)[0x5600587e3b0f] [1,3]:[1a62e574072d:00337] [11] [1,2]:python(_PyEval_EvalFrameDefault+0x30a)[0x558e1351038a] [1,2]:[1a62e574072d:00336] [ 9] python(_PyFunction_FastCallDict+0x11b)[0x558e134e5bab] [1,2]:[1a62e574072d:00336] [10] [1,3]:python(_PyObject_Call_Prepend+0x63)[0x5600587e86a3] [1,3]:[1a62e574072d:00337] [12] python(PyObject_Call+0x3e)[0x5600587e354e] [1,3]:[1a62e574072d:00337] [13] [1,2]:python(_PyObject_FastCallDict+0x26f)[0x558e13464b0f] [1,2]:[1a62e574072d:00336] [11] [1,3]:python(+0x16b50a)[0x56005883d50a] [1,3]:[1a62e574072d:00337] [14] [1,2]:python(_PyObject_Call_Prepend+0x63)[0x558e134696a3] [1,2]:[1a62e574072d:00336] [12] [1,3]:python(_PyEval_EvalFrameDefault+0x877)[0x56005888f8f7] [1,3]:[1a62e574072d:00337] [15] [1,2]:python(PyObject_Call+0x3e)[0x558e1346454e] [1,2]:[1a62e574072d:00336] [13] [1,3]:python(_PyFunction_FastCallDict+0x11b)[0x560058864bab] [1,3]:[1a62e574072d:00337] [16] [1,2]:python(+0x16b50a)[0x558e134be50a] [1,2]:[1a62e574072d:00336] [14] [1,3]:python(_PyObject_FastCallDict+0x26f)[0x5600587e3b0f] [1,3]:[1a62e574072d:00337] [17] python(_PyObject_Call_Prepend+0x63)[0x5600587e86a3] [1,3]:[1a62e574072d:00337] [18] [1,2]:python(_PyEval_EvalFrameDefault+0x877)[0x558e135108f7] [1,2]:[1a62e574072d:00336] [15] python(_PyFunction_FastCallDict+0x11b)[0x558e134e5bab] [1,3]:python(PyObject_Call+0x3e)[0x5600587e354e] [1,3]:[1a62e574072d:00337] [19] [1,2]:[1a62e574072d:00336] [16] python(_PyObject_FastCallDict+0x26f)[0x558e13464b0f] [1,2]:[1a62e574072d:00336] [17] [1,3]:python(+0x16b50a)[0x56005883d50a] [1,3]:[1a62e574072d:00337] [20] [1,2]:python(_PyObject_Call_Prepend+0x63)[0x558e134696a3] [1,2]:[1a62e574072d:00336] [18] [1,3]:python(_PyEval_EvalFrameDefault+0x877)[0x56005888f8f7] [1,3]:[1a62e574072d:00337] [21] [1,2]:python(PyObject_Call+0x3e)[0x558e1346454e] [1,2]:[1a62e574072d:00336] [19] [1,3]:python(+0x19253b)[0x56005886453b] [1,3]:[1a62e574072d:00337] [22] python(+0x198505)[0x56005886a505] [1,3]:[1a62e574072d:00337] [23] [1,2]:python(+0x16b50a)[0x558e134be50a] [1,2]:[1a62e574072d:00336] [20] python(_PyEval_EvalFrameDefault+0x877)[0x558e135108f7] [1,2]:[1a62e574072d:00336] [21] [1,3]:python(_PyEval_EvalFrameDefault+0x30a)[0x56005888f38a] [1,3]:[1a62e574072d:00337] [24] python(+0x191a76)[0x560058863a76] [1,3]:[1a62e574072d:00337] [25] [1,2]:python(+0x19253b)[0x558e134e553b] [1,2]:[1a62e574072d:00336] [22] [1,3]:python(_PyFunction_FastCallDict+0x1bc)[0x560058864c4c] [1,3]:[1a62e574072d:00337] [26] [1,2]:python(+0x198505)[0x558e134eb505] [1,2]:[1a62e574072d:00336] [23] [1,3]:python(_PyObject_FastCallDict+0x26f)[0x5600587e3b0f] [1,3]:[1a62e574072d:00337] [27] [1,2]:python(_PyEval_EvalFrameDefault+0x30a)[0x558e1351038a] [1,2]:[1a62e574072d:00336] [24] python(+0x191a76)[0x558e134e4a76] [1,3]:python(_PyObject_Call_Prepend+0x63)[0x5600587e86a3] [1,3]:[1a62e574072d:00337] [28] [1,2]:[1a62e574072d:00336] [25] python(_PyFunction_FastCallDict+0x1bc)[0x558e134e5c4c] [1,2]:[1a62e574072d:00336] [26] [1,3]:python(PyObject_Call+0x3e)[0x5600587e354e] [1,3]:[1a62e574072d:00337] [29] python(+0x16b50a)[0x56005883d50a] [1,2]:python(_PyObject_FastCallDict+0x26f)[0x558e13464b0f] [1,2]:[1a62e574072d:00336] [27] python(_PyObject_Call_Prepend+0x63)[0x558e134696a3] [1,2]:[1a62e574072d:00336] [28] [1,3]:[1a62e574072d:00337] End of error message [1,2]:python(PyObject_Call+0x3e)[0x558e1346454e] [1,2]:[1a62e574072d:00336] [29] [1,2]:python(+0x16b50a)[0x558e134be50a] [1,2]:[1a62e574072d:00336] End of error message [1,1]:[1a62e574072d:00335] Process received signal [1,1]:[1a62e574072d:00335] Signal: Bus error (7) [1,1]:[1a62e574072d:00335] Signal code: Non-existant physical address (2) [1,1]:[1a62e574072d:00335] Failing at address: 0x7fd41730e00a [1,1]:[1a62e574072d:00335] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd5591ad390] [1,1]:[1a62e574072d:00335] [ 1] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x128e0)[0x7fd44f81f8e0] [1,1]:[1a62e574072d:00335] [1,1]:[ 2] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x12b74)[0x7fd44f81fb74] [1,1]:[1a62e574072d:00335] [ 3] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x14ba5)[0x7fd44f821ba5] [1,1]:[1a62e574072d:00335] [ 4] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(mdb_get+0xbc)[0x7fd44f82240c] [1,1]:[1a62e574072d:00335] [ 5] /opt/conda/lib/python3.6/site-packages/lmdb/cpython.cpython-36m-x86_64-linux-gnu.so(+0x9d9d)[0x7fd44f816d9d] [1,1]:[1a62e574072d:00335] [ 6] [1,1]:python(_PyCFunction_FastCallDict+0x154)[0x55e2b8f02744] [1,1]:[1a62e574072d:00335] [ 7] [1,1]:python(+0x19842c)[0x55e2b8f8942c] [1,1]:[1a62e574072d:00335] [ 8] [1,1]:python(_PyEval_EvalFrameDefault+0x30a)[0x55e2b8fae38a] [1,1]:[1a62e574072d:00335] [ 9] [1,1]:python(_PyFunction_FastCallDict+0x11b)[0x55e2b8f83bab] [1,1]:[1a62e574072d:00335] [10] [1,1]:python(_PyObject_FastCallDict+0x26f)[0x55e2b8f02b0f] [1,1]:[1a62e574072d:00335] [11] [1,1]:python(_PyObject_Call_Prepend+0x63)[0x55e2b8f076a3] [1,1]:[1a62e574072d:00335] [12] [1,1]:python(PyObject_Call+0x3e)[0x55e2b8f0254e] [1,1]:[1a62e574072d:00335] [13] [1,1]:python(+0x16b50a)[0x55e2b8f5c50a] [1,1]:[1a62e574072d:00335] [14] [1,1]:python(_PyEval_EvalFrameDefault+0x877)[0x55e2b8fae8f7] [1,1]:[1a62e574072d:00335] [15] [1,1]:python(_PyFunction_FastCallDict+0x11b)[0x55e2b8f83bab] [1,1]:[1a62e574072d:00335] [16] [1,1]:python(_PyObject_FastCallDict+0x26f)[0x55e2b8f02b0f] [1,1]:[1a62e574072d:00335] [17] [1,1]:python(_PyObject_Call_Prepend+0x63)[0x55e2b8f076a3] [1,1]:[1a62e574072d:00335] [18] [1,1]:python(PyObject_Call+0x3e)[0x55e2b8f0254e] [1,1]:[1a62e574072d:00335] [19] [1,1]:python(+0x16b50a)[0x55e2b8f5c50a] [1,1]:[1a62e574072d:00335] [20] [1,1]:python(_PyEval_EvalFrameDefault+0x877)[0x55e2b8fae8f7] [1,1]:[1a62e574072d:00335] [21] [1,1]:python(+0x19253b)[0x55e2b8f8353b] [1,1]:[1a62e574072d:00335] [22] [1,1]:python(+0x198505)[0x55e2b8f89505] [1,1]:[1a62e574072d:00335] [23] [1,1]:python(_PyEval_EvalFrameDefault+0x30a)[0x55e2b8fae38a] [1,1]:[1a62e574072d:00335] [24] [1,1]:python(+0x191a76)[0x55e2b8f82a76] [1,1]:[1a62e574072d:00335] [25] [1,1]:python(_PyFunction_FastCallDict+0x1bc)[0x55e2b8f83c4c] [1,1]:[1a62e574072d:00335] [26] [1,1]:python(_PyObject_FastCallDict+0x26f)[0x55e2b8f02b0f] [1,1]:[1a62e574072d:00335] [27] [1,1]:python(_PyObject_Call_Prepend+0x63)[0x55e2b8f076a3] [1,1]:[1a62e574072d:00335] [28] [1,1]:python(PyObject_Call+0x3e)[0x55e2b8f0254e] [1,1]:[1a62e574072d:00335] [29] [1,1]:python(+0x16b50a)[0x55e2b8f5c50a] [1,1]:[1a62e574072d:00335] End of error message

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun noticed that process rank 0 with PID 0 on node 1a62e574072d exited on signal 7 (Bus error).