search

gonglinyuan / metro_t0

Code repo for "Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers" (ACL 2023)
20 stars 3 forks source link

Fail to Run Pretraining #1

Open AI-Fatboy opened 10 months ago

AI-Fatboy commented 10 months ago

I followed the instruction to setup the environment and run pretraining of the T5 baseline. I am using 4x A800 GPUs on 2 nodes so I tweaked the script a little bit. Here is the command I am using:

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
fairseq-hydra-train -m --config-dir examples/t5/config/pretraining \
--config-name t5_base_8gpus \
distributed_training.distributed_world_size=8 \
distributed_training.distributed_rank=$(( RANK * 4 )) \
distributed_training.distributed_init_method="tcp://192.168.1.14:44666" \
task.data=/path/to/mywikibook \
hydra.sweep.dir=/path/to/output_003_t5

Here RANK the environment variable is 0 on the first node and 1 on the second node.

The pretraining seems to be stucked after the following logs:

../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprk24gayf, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo1vx1gx5, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprk24gayf, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo1vx1gx5, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprk24gayf, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo1vx1gx5, retrying 198 more times
[2023-08-13 10:57:34,795][fairseq.data.iterators][INFO] - grouped total_num_itrs = 80180
[2023-08-13 10:57:34,799][fairseq.trainer][INFO] - begin training epoch 2
[2023-08-13 10:57:34,800][fairseq_cli.train][INFO] - Start iterating over samples
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpa8nzn28s, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn36xafqb, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpa8nzn28s, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn36xafqb, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpa8nzn28s, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp65tn9jz5, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn36xafqb, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp65tn9jz5, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsro1kc9c, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp65tn9jz5, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpihqj3v92, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsro1kc9c, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpihqj3v92, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsro1kc9c, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
[2023-08-13 10:57:35,455][numexpr.utils][INFO] - Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2023-08-13 10:57:35,456][numexpr.utils][INFO] - Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2023-08-13 10:57:35,456][numexpr.utils][INFO] - NumExpr defaulting to 8 threads.
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpihqj3v92, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpwho59wvg, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7i_f99vn, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv48fb721, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpwho59wvg, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpg9n4lp3t, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7i_f99vn, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7stbg9e3, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv48fb721, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpwho59wvg, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpg9n4lp3t, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7i_f99vn, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7stbg9e3, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv48fb721, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_xhq1bve, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpwho59wvg, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpg9n4lp3t, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7i_f99vn, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7stbg9e3, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv48fb721, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_xhq1bve, retrying 199 more times
../src/plasma/store.cc../src/plasma/store.cc../src/plasma/store.cc:::127412741274: : Allowing the Plasma store to use up to : Allowing the Plasma store to use up to Allowing the Plasma store to use up to 0.462204GB of memory.0.4622040.462204
GB of memory.GB of memory.../src/plasma/store.cc:
1297
../src/plasma/store.cc: :../src/plasma/store.ccStarting object store with directory 1297:/dev/shm: 1297 and huge page support Starting object store with directory : Starting object store with directory /dev/shm/dev/shmdisabled and huge page support  and huge page support 
disableddisabled

../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.462204GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.308136GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_xhq1bve, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.462204GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27xexyt6, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpm3t5h5l6, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27xexyt6, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpm3t5h5l6, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27xexyt6, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.462204GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpm3t5h5l6, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.462204GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpt2f4086i, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqrev4_d7, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp22mdeu4q, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn8zcdjx3, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpt2f4086i, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqrev4_d7, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyp76x2qn, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp22mdeu4q, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn8zcdjx3, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpt2f4086i, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqrev4_d7, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyp76x2qn, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp22mdeu4q, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpn8zcdjx3, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc../src/plasma/store.cc::12741274: : Allowing the Plasma store to use up to Allowing the Plasma store to use up to 0.594368GB of memory.0.594368
GB of memory.../src/plasma/store.cc:
1297../src/plasma/store.cc: :Starting object store with directory 1297/dev/shm:  and huge page support Starting object store with directory /dev/shmdisabled and huge page support 
disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyp76x2qn, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpiu3v5ik0, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxbmo_moi, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpiu3v5ik0, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxbmo_moi, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpiu3v5ik0, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxbmo_moi, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp9f1r06z6, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpaadb9mhc, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp15em8bql, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprnm3112l, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3oaxfj3b, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp9f1r06z6, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpaadb9mhc, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp15em8bql, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprnm3112l, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzo2jf5x9, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3oaxfj3b, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp9f1r06z6, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpaadb9mhc, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp15em8bql, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprnm3112l, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzo2jf5x9, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3oaxfj3b, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpaadb9mhc, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp15em8bql, retrying 197 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprnm3112l, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzo2jf5x9, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpy97kbew0, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3oaxfj3b, retrying 197 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.462204GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvaviv5wx, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpy97kbew0, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvaviv5wx, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpy97kbew0, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvaviv5wx, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp1tjq419a, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpuwdqy464, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp1tjq419a, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7q7m4az9, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjcy09frn, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27_edadr, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplv6l7nw8, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpuwdqy464, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp1tjq419a, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7q7m4az9, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjcy09frn, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27_edadr, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplv6l7nw8, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpuwdqy464, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.594368GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp7q7m4az9, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjcy09frn, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp27_edadr, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplv6l7nw8, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmptnoleq2p, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpc2_y4hg5, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmptnoleq2p, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpc2_y4hg5, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmptnoleq2p, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsus12hjf, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpc2_y4hg5, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsus12hjf, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv0uw6p8s, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdhvfhjli, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpj6lwtwwh, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsus12hjf, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp76p15ofh, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv0uw6p8s, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb87b9lza, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdhvfhjli, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpj6lwtwwh, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp76p15ofh, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv0uw6p8s, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb87b9lza, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdhvfhjli, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpj6lwtwwh, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp76p15ofh, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpv0uw6p8s, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb87b9lza, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.891552GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdhvfhjli, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpj6lwtwwh, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp76p15ofh, retrying 197 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb87b9lza, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqqm4z_za, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp4_3ikzuy, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqqm4z_za, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjwwox8ok, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp4_3ikzuy, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqqm4z_za, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjwwox8ok, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp4_3ikzuy, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpjwwox8ok, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyjpk2a3a, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmps1d5gnw0, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmppcbelm9v, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplyaw5xzp, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxvx2xl5g, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyjpk2a3a, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmps1d5gnw0, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmppcbelm9v, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplyaw5xzp, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxvx2xl5g, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpyjpk2a3a, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmps1d5gnw0, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmppcbelm9v, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplyaw5xzp, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_0bu499n, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpxvx2xl5g, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.576898GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdunz7mkl, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp98v48lsg, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_0bu499n, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdunz7mkl, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp98v48lsg, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_0bu499n, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdunz7mkl, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp98v48lsg, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc../src/plasma/store.cc::12741274: : Allowing the Plasma store to use up to Allowing the Plasma store to use up to 0.2069380.206938GB of memory.GB of memory.

../src/plasma/store.cc../src/plasma/store.cc::12971297: : Starting object store with directory Starting object store with directory /dev/shm/dev/shm and huge page support  and huge page support disableddisabled

../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5j550t5e, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvhj403ka, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpifz2dlz1, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5j550t5e, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp42b62hz3, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_3y02k40, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvhj403ka, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpifz2dlz1, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5j550t5e, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp42b62hz3, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_3y02k40, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpvhj403ka, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdi3xuw9l, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpifz2dlz1, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpbxlw3p_r, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp42b62hz3, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp_3y02k40, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw6jqztgq, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdi3xuw9l, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.865347GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpbxlw3p_r, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw6jqztgq, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpdi3xuw9l, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpbxlw3p_r, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw6jqztgq, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpnktfo0dm, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqon7mc6v, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpnktfo0dm, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpz9yvu6ak, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqon7mc6v, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3z517802, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpnktfo0dm, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpz9yvu6ak, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpqon7mc6v, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpmicmyzhq, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3z517802, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpz9yvu6ak, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsxflpclu, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpkvl78rmd, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpmicmyzhq, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo0dw02tb, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp3z517802, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsxflpclu, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpkvl78rmd, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpmicmyzhq, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.206938GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo0dw02tb, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpsxflpclu, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpkvl78rmd, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpo0dw02tb, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplnlmm3io, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5z7hg8y9, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp97yitntc, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplnlmm3io, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5z7hg8y9, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp97yitntc, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplnlmm3io, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5z7hg8y9, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw30i3mb8, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp97yitntc, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmplnlmm3io, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp5z7hg8y9, retrying 197 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw30i3mb8, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzjsnhnsj, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp639k100u, retrying 200 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb54virj3, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpw30i3mb8, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzjsnhnsj, retrying 199 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.310407GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp639k100u, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb54virj3, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpzjsnhnsj, retrying 198 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmp639k100u, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmpb54virj3, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmphq1j01h9, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmphq1j01h9, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmphq1j01h9, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.502836GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprehp5o38, retrying 200 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprehp5o38, retrying 199 more times
../src/plasma/io.cc:177: Connection to IPC socket failed for pathname /tmp/tmprehp5o38, retrying 198 more times
../src/plasma/store.cc:1274: Allowing the Plasma store to use up to 0.754253GB of memory.
../src/plasma/store.cc:1297: Starting object store with directory /dev/shm and huge page support disabled
gonglinyuan commented 10 months ago

Thanks for the detailed logs. We're currently working on the issue you've described. We'll update this thread as soon as we have insights or a fix.

jph00 commented 4 months ago

@gonglinyuan any update on this?