search

microsoft / MeshTransformer

Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"
https://arxiv.org/abs/2012.09760
MIT License
614 stars 95 forks source link

Questions about training on 3DPW #38

Closed wjingdan closed 3 years ago

wjingdan commented 3 years ago

Hello. Thanks for your great work. I have some questions about training on 3DPW. According to docs/EXP.md, i fine-tune your pre-trained model (metro_h36m_state_dict.bin) on 3DPW training set. But the result is not like what you described in the paper. How did you train on the 3DPW data set?

image

kevinlin311tw commented 3 years ago

You may find our original training log for 3DPW as below.

Note: (1) At that time, we were trying to fine-tune for 30 epochs. But we found we already got good results at the early epoch, so that we terminated the training and didn't tune it further. Below, it is the training log from 1st to 5th epoch. We got good result at 4th epoch. (2) We have refactored the codebase before code release. The following log is from the old codebase. That's why the log style is a bit different. 

2020-11-08 20:25:40 [1,0]<stderr>:11-08 20:25:39.515 az-scus-v100-2-worker-mgfopa 573 aml_main.py:20    cmd_run(): start to cmd run:
2020-11-08 20:25:40 [1,0]<stderr>:python -m torch.distributed.launch --nproc_per_node=8 tools/human_mesh/run_train_simplifiedmesh_2d3d_mvm_newbone_conv.py --train_yaml 3dpw_backup/train.yaml --val_yaml 3dpw_backup/test_has_gender.yaml --arch hrnet-w64 --model_name_or_path models/captioning/bert-base-uncased/ --num_workers 2 --logging_steps 20 --resume_checkpoint _output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin --per_gpu_train_batch_size 30 --per_gpu_eval_batch_size 30 --num_hidden_layers 4 --num_attention_heads 4 --lr 1e-4 --backbone_pretrained --fix_backbone 0 --object_query 1 --masking_inputs 0 --img_scale_factor 1 --scheduler iter_step --num_train_epochs 30 --input_feat_dim 2051,512,128 --hidden_feat_dim 1024,256,128 --vertices_loss_weight 100.0 --joints_loss_weight 1000.0 --vloss_w_full 0.33 --vloss_w_sub 0.33 --vloss_w_sub2 0.33
2020-11-08 20:25:40 [1,0]<stderr>:
2020-11-08 20:25:40 [1,0]<stdout>:*****************************************
2020-11-08 20:25:40 [1,0]<stdout>:Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
2020-11-08 20:25:40 [1,0]<stdout>:*****************************************
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 5
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 2
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 4
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 3
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 1
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 7
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 6
2020-11-08 20:25:43 [1,0]<stdout>:Init distributed training on local rank 0
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:990 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:NCCL version 2.4.8+cuda10.1
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:995 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:996 [6] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:992 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:991 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:993 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:994 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
2020-11-08 20:25:49 [1,0]<stdout>:
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:997 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0>
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0.
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 00 :    0   1   2   3   7   4   6   5
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 01 :    0   2   4   5   7   6   1   3
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 02 :    0   3   1   6   7   5   4   2
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 03 :    0   3   2   1   6   4   7   5
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 04 :    0   5   6   4   7   3   2   1
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 05 :    0   5   7   4   6   1   2   3
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 06 :    0   1   2   3   7   4   6   5
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 07 :    0   2   4   5   7   6   1   3
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 08 :    0   3   1   6   7   5   4   2
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 09 :    0   3   2   1   6   4   7   5
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 10 :    0   5   6   4   7   3   2   1
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 11 :    0   5   7   4   6   1   2   3
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 00 : 1[2] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 00 : 3[7] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 00 : 4[0] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 00 : 2[1] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 00 : 7[6] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 00 : 5[4] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 00 : 6[5] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 00 : 0[3] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 01 : 1[2] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 01 : 3[7] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 01 : 4[0] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 01 : 2[1] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 01 : 7[6] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 01 : 5[4] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 01 : 6[5] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 01 : 0[3] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 02 : 1[2] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 02 : 3[7] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 02 : 4[0] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 02 : 7[6] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 02 : 2[1] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 02 : 0[3] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 02 : 6[5] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 02 : 5[4] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 03 : 1[2] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 03 : 3[7] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 03 : 4[0] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 03 : 7[6] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 03 : 2[1] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 03 : 0[3] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 03 : 5[4] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 03 : 6[5] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 04 : 3[7] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 04 : 1[2] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 04 : 4[0] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 04 : 7[6] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 04 : 2[1] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 04 : 0[3] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 04 : 5[4] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 04 : 6[5] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 05 : 1[2] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 05 : 3[7] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 05 : 4[0] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 05 : 7[6] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 05 : 2[1] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 05 : 0[3] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 05 : 6[5] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 05 : 5[4] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 06 : 3[7] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 06 : 4[0] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 06 : 2[1] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 06 : 1[2] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 06 : 7[6] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 06 : 0[3] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 06 : 6[5] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 06 : 5[4] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 07 : 3[7] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 07 : 4[0] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 07 : 2[1] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 07 : 1[2] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 07 : 7[6] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 07 : 0[3] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 07 : 6[5] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 07 : 5[4] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 08 : 3[7] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 08 : 4[0] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 08 : 2[1] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 08 : 1[2] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 08 : 7[6] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 08 : 0[3] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 08 : 6[5] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 08 : 5[4] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 09 : 3[7] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 09 : 2[1] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 09 : 4[0] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 09 : 1[2] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 09 : 7[6] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 09 : 0[3] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 09 : 6[5] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 09 : 5[4] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 10 : 3[7] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 10 : 2[1] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 10 : 4[0] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 10 : 1[2] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 10 : 7[6] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 10 : 0[3] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 10 : 6[5] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 10 : 5[4] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 11 : 3[7] -> 0[3] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 11 : 1[2] -> 2[1] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 11 : 4[0] -> 6[5] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 11 : 7[6] -> 4[0] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 11 : 2[1] -> 3[7] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 11 : 6[5] -> 1[2] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 11 : 0[3] -> 5[4] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 11 : 5[4] -> 7[6] via P2P/IPC
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO comm 0x7f73e4001d50 rank 3 nranks 8 cudaDev 3 nvmlDev 7 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO comm 0x7f8cd0001d50 rank 4 nranks 8 cudaDev 4 nvmlDev 0 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO comm 0x7f43d8001d50 rank 7 nranks 8 cudaDev 7 nvmlDev 6 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO comm 0x7f47d8001d50 rank 1 nranks 8 cudaDev 1 nvmlDev 2 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO comm 0x7f64f4001d50 rank 2 nranks 8 cudaDev 2 nvmlDev 1 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO comm 0x7f7d08001d50 rank 6 nranks 8 cudaDev 6 nvmlDev 5 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO comm 0x7f79e8001d50 rank 0 nranks 8 cudaDev 0 nvmlDev 3 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO comm 0x7fb7a8001d50 rank 5 nranks 8 cudaDev 5 nvmlDev 4 - Init COMPLETE
2020-11-08 20:25:49 [1,0]<stdout>:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO Launch mode Parallel
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:49 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:49 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:52 [1,0]<stdout>:2020-11-08 20:25:50,871 Mesh regression INFO: Using 8 GPUs
2020-11-08 20:25:52 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:52 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:55 [1,0]<stderr>:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
2020-11-08 20:25:55 [1,0]<stderr>:  self._set_intXint(row, col, x.flat[0])
2020-11-08 20:25:58 [1,0]<stdout>:2020-11-08 20:25:57,499 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4
2020-11-08 20:25:58 [1,0]<stdout>:2020-11-08 20:25:57,759 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,410 Mesh regression INFO: Init model from scratch.
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,459 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,509 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,849 Mesh regression INFO: Init model from scratch.
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,898 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:25:59,946 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4
2020-11-08 20:26:01 [1,0]<stdout>:2020-11-08 20:26:00,139 Mesh regression INFO: Init model from scratch.
2020-11-08 20:26:08 [1,0]<stdout>:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2020-11-08 20:26:08 [1,0]<stdout>:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2020-11-08 20:26:08 [1,0]<stdout>:
2020-11-08 20:26:08 [1,0]<stdout>:
2020-11-08 20:26:08 [1,0]<stdout>:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2020-11-08 20:26:08 [1,0]<stdout>:2020-11-08 20:26:07,957 Mesh regression INFO: => loading hrnet-v2-w64 model
2020-11-08 20:26:08 [1,0]<stdout>:2020-11-08 20:26:08,007 Mesh regression INFO: Transformer Encoder 2 total parameters: 102256646
2020-11-08 20:26:08 [1,0]<stdout>:2020-11-08 20:26:08,072 Mesh regression INFO: Backbone model total parameters: 128059944
2020-11-08 20:26:08 [1,0]<stdout>:2020-11-08 20:26:08,189 Mesh regression INFO: Loading state dict from checkpoint _output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:2020-11-08 20:26:18,607 Mesh regression INFO: Training parameters Namespace(adam_epsilon=1e-08, arch='hrnet-w64', backbone_pretrained=True, config_name='', data_dir='datasets', device=device(type='cuda'), distributed=True, do_lower_case=False, drop_out=0.1, effective_batch_size=-1, fix_backbone=0, hidden_feat_dim='1024,256,128', hidden_size=-1, img_feature_dim=2051, img_scale_factor=1, input_feat_dim='2051,512,128', intermediate_size=-1, joints_loss_weight=1000.0, load_partial_weights=False, local_rank=0, logging_steps=20, lr=0.0001, mask_prob=0.15, mask_type='bidirectional', masking_inputs=0, max_masked_tokens=3, model_name_or_path='models/captioning/bert-base-uncased/', momentum=0.9, no_sort_by_conf=False, num_attention_heads=4, num_gpus=8, num_hidden_layers=4, num_train_epochs=30, num_workers=2, object_query=1, on_memory=False, output_dir='_keli/output/', per_gpu_eval_batch_size=30, per_gpu_train_batch_size=30, resume_checkpoint='_output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin', run_eval_only=False, save_steps=50000, scheduler='iter_step', seed=88, tokenizer_name='', train_yaml='3dpw_backup/train.yaml', val_yaml='3dpw_backup/test_has_gender.yaml', val_yaml2='imagenet2012/test.yaml', val_yaml3='imagenet2012/test.yaml', vertices_loss_weight=100.0, vloss_w_full=0.33, vloss_w_sub=0.33, vloss_w_sub2=0.33, warmup_steps=0, weight_decay=0.05)
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/train.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/test_has_gender.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/test_has_gender.yaml
2020-11-08 20:26:20 [1,0]<stdout>:3dpw_backup/test_has_gender.yaml
2020-11-08 20:22020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 10.9480, 2d joint loss: 0.0066, 3d joint loss: 0.0066, vertex loss: 0.0373, compute: 4.2248, data: 2.8355, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:08 epoch: 0 iter: 20 max mem : 27601  loss: 11.1114, 2d joint loss: 0.0063, 3d joint loss: 0.0067, vertex loss: 0.0382, compute: 4.2246, data: 2.8432, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stdout>:2020-11-08 20:27:44,999 Mesh regression INFO: eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 12.2818, 2d joint loss: 0.0067, 3d joint loss: 0.0076, vertex loss: 0.0396, compute: 4.2247, data: 2.8407, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 10.8245, 2d joint loss: 0.0062, 3d joint loss: 0.0064, vertex loss: 0.0380, compute: 4.2249, data: 2.8416, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 11.5213, 2d joint loss: 0.0067, 3d joint loss: 0.0070, vertex loss: 0.0390, compute: 4.2248, data: 2.8314, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 11.7696, 2d joint loss: 0.0065, 3d joint loss: 0.0072, vertex loss: 0.0392, compute: 4.2249, data: 2.8450, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:10 epoch: 0 iter: 20 max mem : 27601  loss: 11.1805, 2d joint loss: 0.0062, 3d joint loss: 0.0067, vertex loss: 0.0389, compute: 4.2253, data: 2.8389, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:11 epoch: 0 iter: 20 max mem : 27601  loss: 11.0279, 2d joint loss: 0.0059, 3d joint loss: 0.0066, vertex loss: 0.0388, compute: 4.2254, data: 2.8384, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601  loss: 12.2818, 2d joint loss: 0.0067, 3d joint loss: 0.0076, vertex loss: 0.0396, compute: 4.2247, data: 2.8407, lr: 0.000100
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:27:47 [1,0]<stderr>:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
2020-11-08 20:27:47 [1,0]<stderr>:  if np.issubdtype(image.dtype, np.float):
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:11 epoch: 0 iter: 40 max mem : 27601  loss: 9.1141, 2d joint loss: 0.0048, 3d joint loss: 0.0052, vertex loss: 0.0342, compute: 2.8099, data: 1.4785, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.0047, 2d joint loss: 0.0052, 3d joint loss: 0.0052, vertex loss: 0.0332, compute: 2.8102, data: 1.4750, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stdout>:2020-11-08 20:28:12,912 Mesh regression INFO: eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.3616, 2d joint loss: 0.0048, 3d joint loss: 0.0055, vertex loss: 0.0341, compute: 2.8102, data: 1.5135, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.2867, 2d joint loss: 0.0052, 3d joint loss: 0.0053, vertex loss: 0.0343, compute: 2.8103, data: 1.4741, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.4480, 2d joint loss: 0.0053, 3d joint loss: 0.0055, vertex loss: 0.0345, compute: 2.8103, data: 1.4804, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.3447, 2d joint loss: 0.0051, 3d joint loss: 0.0054, vertex loss: 0.0343, compute: 2.8103, data: 1.4807, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:13 epoch: 0 iter: 40 max mem : 27601  loss: 8.8068, 2d joint loss: 0.0047, 3d joint loss: 0.0050, vertex loss: 0.0338, compute: 2.8104, data: 1.4793, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:13 epoch: 0 iter: 40 max mem : 27601  loss: 8.7020, 2d joint loss: 0.0046, 3d joint loss: 0.0049, vertex loss: 0.0336, compute: 2.8105, data: 1.4803, lr: 0.000100
2020-11-08 20:28:14 [1,0]<stderr>:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601  loss: 9.3616, 2d joint loss: 0.0048, 3d joint loss: 0.0055, vertex loss: 0.0341, compute: 2.8102, data: 1.5135, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 7.8472, 2d joint loss: 0.0044, 3d joint loss: 0.0043, vertex loss: 0.0307, compute: 2.3186, data: 1.0205, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stdout>:2020-11-08 20:28:39,616 Mesh regression INFO: eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 8.2019, 2d joint loss: 0.0041, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3185, data: 1.0535, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 7.9426, 2d joint loss: 0.0041, 3d joint loss: 0.0044, vertex loss: 0.0314, compute: 2.3186, data: 1.0238, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 8.0331, 2d joint loss: 0.0043, 3d joint loss: 0.0045, vertex loss: 0.0315, compute: 2.3186, data: 1.0201, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 7.5853, 2d joint loss: 0.0040, 3d joint loss: 0.0041, vertex loss: 0.0309, compute: 2.3186, data: 1.0239, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 8.2093, 2d joint loss: 0.0043, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3187, data: 1.0245, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 8.2107, 2d joint loss: 0.0046, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3187, data: 1.0244, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 7.8058, 2d joint loss: 0.0041, 3d joint loss: 0.0043, vertex loss: 0.0313, compute: 2.3188, data: 1.0236, lr: 0.000100
2020-11-08 20:28:41 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601  loss: 8.2019, 2d joint loss: 0.0041, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3185, data: 1.0535, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.1706, 2d joint loss: 0.0040, 3d joint loss: 0.0039, vertex loss: 0.0291, compute: 2.0746, data: 0.7931, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stdout>:2020-11-08 20:29:06,469 Mesh regression INFO: eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.4384, 2d joint loss: 0.0037, 3d joint loss: 0.0041, vertex loss: 0.0298, compute: 2.0745, data: 0.8243, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.2779, 2d joint loss: 0.0037, 3d joint loss: 0.0039, vertex loss: 0.0298, compute: 2.0746, data: 0.7959, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.3775, 2d joint loss: 0.0038, 3d joint loss: 0.0040, vertex loss: 0.0296, compute: 2.0746, data: 0.7962, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.3226, 2d joint loss: 0.0039, 3d joint loss: 0.0040, vertex loss: 0.0297, compute: 2.0746, data: 0.7929, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 6.9406, 2d joint loss: 0.0036, 3d joint loss: 0.0037, vertex loss: 0.0292, compute: 2.0747, data: 0.7958, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.1073, 2d joint loss: 0.0037, 3d joint loss: 0.0038, vertex loss: 0.0294, compute: 2.0748, data: 0.7959, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:45 epoch: 0 iter: 80 max mem : 27601  loss: 7.6323, 2d joint loss: 0.0043, 3d joint loss: 0.0042, vertex loss: 0.0299, compute: 2.0750, data: 0.7958, lr: 0.000100
2020-11-08 20:29:08 [1,0]<stderr>:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601  loss: 7.4384, 2d joint loss: 0.0037, 3d joint loss: 0.0041, vertex loss: 0.0298, compute: 2.0745, data: 0.8243, lr: 0.000100
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stdout>:2020-11-08 20:32:19,937 Mesh regression INFO: Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:20 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 1  mPVE:  80.65, mPJPE:  76.19, mPVE_smpl:  90.18, mPJPE_smpl:  76.88, PAmPJPE_smpl:  48.04, Data Count: 35520.00
2020-11-08 20:32:26 [1,0]<stdout>:2020-11-08 20:32:26,475 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-1-94
2020-11-08 20:32:26 [1,0]<stderr>:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-1-94
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.7488, 2d joint loss: 0.0034, 3d joint loss: 0.0036, vertex loss: 0.0283, compute: 3.7296, data: 2.4009, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 7.0071, 2d joint loss: 0.0040, 3d joint loss: 0.0038, vertex loss: 0.0284, compute: 3.7296, data: 2.4002, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.7134, 2d joint loss: 0.0037, 3d joint loss: 0.0035, vertex loss: 0.0280, compute: 3.7297, data: 2.3979, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.6355, 2d joint loss: 0.0034, 3d joint loss: 0.0035, vertex loss: 0.0281, compute: 3.7297, data: 2.3990, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stdout>:2020-11-08 20:32:33,473 Mesh regression INFO: eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.8787, 2d joint loss: 0.0034, 3d joint loss: 0.0037, vertex loss: 0.0284, compute: 3.7297, data: 2.4923, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.6848, 2d joint loss: 0.0035, 3d joint loss: 0.0035, vertex loss: 0.0281, compute: 3.7297, data: 2.3968, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.5730, 2d joint loss: 0.0034, 3d joint loss: 0.0034, vertex loss: 0.0282, compute: 3.7297, data: 2.4002, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.9428, 2d joint loss: 0.0036, 3d joint loss: 0.0037, vertex loss: 0.0285, compute: 3.7298, data: 2.3993, lr: 0.000100
2020-11-08 20:32:35 [1,0]<stderr>:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601  loss: 6.8787, 2d joint loss: 0.0034, 3d joint loss: 0.0037, vertex loss: 0.0284, compute: 3.7297, data: 2.4923, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.2608, 2d joint loss: 0.0034, 3d joint loss: 0.0032, vertex loss: 0.0268, compute: 3.3308, data: 2.0158, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.5182, 2d joint loss: 0.0037, 3d joint loss: 0.0034, vertex loss: 0.0272, compute: 3.3308, data: 2.0188, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stdout>:2020-11-08 20:33:00,197 Mesh regression INFO: eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.3791, 2d joint loss: 0.0031, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3308, data: 2.0993, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.2357, 2d joint loss: 0.0032, 3d joint loss: 0.0032, vertex loss: 0.0270, compute: 3.3308, data: 2.0167, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.4800, 2d joint loss: 0.0033, 3d joint loss: 0.0034, vertex loss: 0.0274, compute: 3.3308, data: 2.0169, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.1705, 2d joint loss: 0.0032, 3d joint loss: 0.0031, vertex loss: 0.0271, compute: 3.3308, data: 2.0192, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.2436, 2d joint loss: 0.0032, 3d joint loss: 0.0032, vertex loss: 0.0270, compute: 3.3308, data: 2.0159, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.2898, 2d joint loss: 0.0032, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3309, data: 2.0196, lr: 0.000100
2020-11-08 20:33:02 [1,0]<stderr>:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601  loss: 6.3791, 2d joint loss: 0.0031, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3308, data: 2.0993, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stdout>:2020-11-08 20:33:27,007 Mesh regression INFO: eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 6.0413, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0262, compute: 3.0464, data: 1.8175, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 6.0003, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0263, compute: 3.0465, data: 1.7471, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 6.1838, 2d joint loss: 0.0035, 3d joint loss: 0.0032, vertex loss: 0.0264, compute: 3.0465, data: 1.7499, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 5.9973, 2d joint loss: 0.0033, 3d joint loss: 0.0031, vertex loss: 0.0260, compute: 3.0465, data: 1.7436, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 5.9437, 2d joint loss: 0.0031, 3d joint loss: 0.0030, vertex loss: 0.0261, compute: 3.0465, data: 1.7455, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 5.8686, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0262, compute: 3.0465, data: 1.7466, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 5.9656, 2d joint loss: 0.0031, 3d joint loss: 0.0030, vertex loss: 0.0262, compute: 3.0465, data: 1.7437, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 6.0839, 2d joint loss: 0.0031, 3d joint loss: 0.0031, vertex loss: 0.0263, compute: 3.0465, data: 1.7471, lr: 0.000100
2020-11-08 20:33:29 [1,0]<stderr>:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601  loss: 6.0413, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0262, compute: 3.0464, data: 1.8175, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.8663, 2d joint loss: 0.0033, 3d joint loss: 0.0030, vertex loss: 0.0256, compute: 2.8316, data: 1.5449, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.7580, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0256, compute: 2.8316, data: 1.5426, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.6847, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.5394, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stdout>:2020-11-08 20:33:53,572 Mesh regression INFO: eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.7673, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.6062, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.6910, 2d joint loss: 0.0031, 3d joint loss: 0.0029, vertex loss: 0.0252, compute: 2.8317, data: 1.5392, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.7129, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.5409, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.6514, 2d joint loss: 0.0029, 3d joint loss: 0.0028, vertex loss: 0.0255, compute: 2.8317, data: 1.5419, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.7965, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0255, compute: 2.8318, data: 1.5424, lr: 0.000100
2020-11-08 20:33:53 [1,0]<stderr>:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601  loss: 5.7673, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.6062, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601  loss: 5.4878, 2d joint loss: 0.0029, 3d joint loss: 0.0027, vertex loss: 0.0248, compute: 2.6734, data: 1.3818, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601  loss: 5.6730, 2d joint loss: 0.0032, 3d joint loss: 0.0029, vertex loss: 0.0249, compute: 2.6734, data: 1.3874, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stdout>:2020-11-08 20:34:21,727 Mesh regression INFO: eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601  loss: 5.5217, 2d joint loss: 0.0027, 3d joint loss: 0.0028, vertex loss: 0.0247, compute: 2.6734, data: 1.4504, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601  loss: 5.4529, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0247, compute: 2.6735, data: 1.3806, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601  loss: 5.4541, 2d joint loss: 0.0029, 3d joint loss: 0.0027, vertex loss: 0.0246, compute: 2.6735, data: 1.3804, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601  loss: 5.5094, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0249, compute: 2.6735, data: 1.3835, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601  loss: 5.5450, 2d joint loss: 0.0028, 3d joint loss: 0.0028, vertex loss: 0.0249, compute: 2.6735, data: 1.3832, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601  loss: 5.4613, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0249, compute: 2.6736, data: 1.3828, lr: 0.000100
2020-11-08 20:34:23 [1,0]<stderr>:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601  loss: 5.5217, 2d joint loss: 0.0027, 3d joint loss: 0.0028, vertex loss: 0.0247, compute: 2.6734, data: 1.4504, lr: 0.000100
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stdout>:2020-11-08 20:35:51,460 Mesh regression INFO: Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:35:54 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 2  mPVE:  81.17, mPJPE:  76.61, mPVE_smpl:  90.06, mPJPE_smpl:  77.41, PAmPJPE_smpl:  48.42, Data Count: 35520.00
2020-11-08 20:36:03 [1,0]<stdout>:2020-11-08 20:36:00,965 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-2-188
2020-11-08 20:36:03 [1,0]<stderr>:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-2-188
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601  loss: 5.3231, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0243, compute: 2.9736, data: 1.6445, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601  loss: 5.2614, 2d joint loss: 0.0029, 3d joint loss: 0.0026, vertex loss: 0.0240, compute: 2.9737, data: 1.6414, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601  loss: 5.4484, 2d joint loss: 0.0030, 3d joint loss: 0.0027, vertex loss: 0.0244, compute: 2.9736, data: 1.6495, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601  loss: 5.2795, 2d joint loss: 0.0028, 3d joint loss: 0.0026, vertex loss: 0.0242, compute: 2.9737, data: 1.6426, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601  loss: 5.3268, 2d joint loss: 0.0028, 3d joint loss: 0.0026, vertex loss: 0.0243, compute: 2.9737, data: 1.6415, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601  loss: 5.3337, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0243, compute: 2.9737, data: 1.6439, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601  loss: 5.2492, 2d joint loss: 0.0027, 3d joint loss: 0.0025, vertex loss: 0.0243, compute: 2.9737, data: 1.6435, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stdout>:2020-11-08 20:36:15,252 Mesh regression INFO: eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601  loss: 5.3138, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0241, compute: 2.9737, data: 1.7541, lr: 0.000100
2020-11-08 20:36:18 [1,0]<stderr>:INFO:Mesh regression:eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601  loss: 5.3138, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0241, compute: 2.9737, data: 1.7541, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.2427, 2d joint loss: 0.0029, 3d joint loss: 0.0026, vertex loss: 0.0238, compute: 2.8244, data: 1.5096, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.1227, 2d joint loss: 0.0026, 3d joint loss: 0.0025, vertex loss: 0.0238, compute: 2.8244, data: 1.5054, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.1471, 2d joint loss: 0.0027, 3d joint loss: 0.0025, vertex loss: 0.0237, compute: 2.8244, data: 1.5045, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stdout>:2020-11-08 20:36:41,872 Mesh regression INFO: eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.1901, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0237, compute: 2.8244, data: 1.6065, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.0998, 2d joint loss: 0.0028, 3d joint loss: 0.0025, vertex loss: 0.0235, compute: 2.8244, data: 1.5023, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.0865, 2d joint loss: 0.0027, 3d joint loss: 0.0025, vertex loss: 0.0236, compute: 2.8244, data: 1.5033, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.1302, 2d joint loss: 0.0027, 3d joint loss: 0.0025, vertex loss: 0.0237, compute: 2.8244, data: 1.5025, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.0928, 2d joint loss: 0.0027, 3d joint loss: 0.0024, vertex loss: 0.0238, compute: 2.8244, data: 1.5037, lr: 0.000100
2020-11-08 20:36:42 [1,0]<stderr>:INFO:Mesh regression:eta: 2:02:23 epoch: 2 iter: 220 max mem : 27601  loss: 5.1901, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0237, compute: 2.8244, data: 1.6065, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 5.0558, 2d joint loss: 0.0028, 3d joint loss: 0.0024, vertex loss: 0.0233, compute: 2.6999, data: 1.3929, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 4.9513, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0234, compute: 2.6999, data: 1.3875, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stdout>:2020-11-08 20:37:08,490 Mesh regression INFO: eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 5.0476, 2d joint loss: 0.0026, 3d joint loss: 0.0025, vertex loss: 0.0232, compute: 2.6999, data: 1.4832, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 4.9245, 2d joint loss: 0.0026, 3d joint loss: 0.0023, vertex loss: 0.0232, compute: 2.6999, data: 1.3866, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 4.9570, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0232, compute: 2.6999, data: 1.3860, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 5.0122, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0233, compute: 2.6999, data: 1.3878, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 4.9748, 2d joint loss: 0.0027, 3d joint loss: 0.0024, vertex loss: 0.0231, compute: 2.7000, data: 1.3857, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 4.9565, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0233, compute: 2.7000, data: 1.3887, lr: 0.000100
2020-11-08 20:37:09 [1,0]<stderr>:INFO:Mesh regression:eta: 1:56:05 epoch: 2 iter: 240 max mem : 27601  loss: 5.0476, 2d joint loss: 0.0026, 3d joint loss: 0.0025, vertex loss: 0.0232, compute: 2.6999, data: 1.4832, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.8369, 2d joint loss: 0.0027, 3d joint loss: 0.0023, vertex loss: 0.0227, compute: 2.5946, data: 1.2876, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.9431, 2d joint loss: 0.0028, 3d joint loss: 0.0024, vertex loss: 0.0229, compute: 2.5946, data: 1.2937, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stdout>:2020-11-08 20:37:35,109 Mesh regression INFO: eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.8891, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0228, compute: 2.5946, data: 1.3788, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.7978, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0228, compute: 2.5946, data: 1.2879, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.8540, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0228, compute: 2.5946, data: 1.2895, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.7929, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0228, compute: 2.5946, data: 1.2883, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.8004, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0228, compute: 2.5946, data: 1.2904, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.7910, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0229, compute: 2.5946, data: 1.2892, lr: 0.000100
2020-11-08 20:37:36 [1,0]<stderr>:INFO:Mesh regression:eta: 1:50:42 epoch: 2 iter: 260 max mem : 27601  loss: 4.8891, 2d joint loss: 0.0026, 3d joint loss: 0.0024, vertex loss: 0.0228, compute: 2.5946, data: 1.3788, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.7174, 2d joint loss: 0.0026, 3d joint loss: 0.0022, vertex loss: 0.0223, compute: 2.5044, data: 1.2035, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.6777, 2d joint loss: 0.0025, 3d joint loss: 0.0022, vertex loss: 0.0224, compute: 2.5043, data: 1.2042, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.8158, 2d joint loss: 0.0027, 3d joint loss: 0.0023, vertex loss: 0.0226, compute: 2.5043, data: 1.2092, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stdout>:2020-11-08 20:38:01,727 Mesh regression INFO: eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.7779, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0224, compute: 2.5044, data: 1.2895, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.6983, 2d joint loss: 0.0025, 3d joint loss: 0.0022, vertex loss: 0.0225, compute: 2.5044, data: 1.2050, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.6773, 2d joint loss: 0.0024, 3d joint loss: 0.0022, vertex loss: 0.0224, compute: 2.5044, data: 1.2063, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.7378, 2d joint loss: 0.0025, 3d joint loss: 0.0022, vertex loss: 0.0224, compute: 2.5044, data: 1.2052, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.7000, 2d joint loss: 0.0025, 3d joint loss: 0.0022, vertex loss: 0.0224, compute: 2.5044, data: 1.2037, lr: 0.000100
2020-11-08 20:38:03 [1,0]<stderr>:INFO:Mesh regression:eta: 1:46:01 epoch: 2 iter: 280 max mem : 27601  loss: 4.7779, 2d joint loss: 0.0025, 3d joint loss: 0.0023, vertex loss: 0.0224, compute: 2.5044, data: 1.2895, lr: 0.000100
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stdout>:2020-11-08 20:39:24,942 Mesh regression INFO: Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:27 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 3  mPVE:  81.12, mPJPE:  76.83, mPVE_smpl:  88.76, mPJPE_smpl:  77.20, PAmPJPE_smpl:  48.09, Data Count: 35520.00
2020-11-08 20:39:36 [1,0]<stdout>:2020-11-08 20:39:34,695 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-3-282
2020-11-08 20:39:36 [1,0]<stderr>:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-3-282
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.6065, 2d joint loss: 0.0025, 3d joint loss: 0.0022, vertex loss: 0.0220, compute: 2.7189, data: 1.3889, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.6905, 2d joint loss: 0.0026, 3d joint loss: 0.0022, vertex loss: 0.0222, compute: 2.7190, data: 1.3942, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.5593, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0221, compute: 2.7190, data: 1.3917, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stdout>:2020-11-08 20:39:56,193 Mesh regression INFO: eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.6461, 2d joint loss: 0.0024, 3d joint loss: 0.0022, vertex loss: 0.0220, compute: 2.7190, data: 1.5060, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.5758, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0220, compute: 2.7190, data: 1.3892, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.5576, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0220, compute: 2.7190, data: 1.3896, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.6297, 2d joint loss: 0.0024, 3d joint loss: 0.0022, vertex loss: 0.0221, compute: 2.7190, data: 1.3906, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.5822, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0222, compute: 2.7190, data: 1.3904, lr: 0.000100
2020-11-08 20:39:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:54:11 epoch: 3 iter: 300 max mem : 27601  loss: 4.6461, 2d joint loss: 0.0024, 3d joint loss: 0.0022, vertex loss: 0.0220, compute: 2.7190, data: 1.5060, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.4533, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0217, compute: 2.6323, data: 1.3117, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.4590, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0217, compute: 2.6323, data: 1.3093, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.4961, 2d joint loss: 0.0025, 3d joint loss: 0.0021, vertex loss: 0.0217, compute: 2.6323, data: 1.3090, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stdout>:2020-11-08 20:40:22,845 Mesh regression INFO: eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.5247, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0217, compute: 2.6323, data: 1.4201, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.5701, 2d joint loss: 0.0026, 3d joint loss: 0.0021, vertex loss: 0.0218, compute: 2.6323, data: 1.3140, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.4618, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0217, compute: 2.6323, data: 1.3094, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.4817, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0218, compute: 2.6323, data: 1.3105, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.5216, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0218, compute: 2.6323, data: 1.3107, lr: 0.000100
2020-11-08 20:40:24 [1,0]<stderr>:INFO:Mesh regression:eta: 1:49:40 epoch: 3 iter: 320 max mem : 27601  loss: 4.5247, 2d joint loss: 0.0024, 3d joint loss: 0.0021, vertex loss: 0.0217, compute: 2.6323, data: 1.4201, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.4620, 2d joint loss: 0.0025, 3d joint loss: 0.0021, vertex loss: 0.0214, compute: 2.5558, data: 1.2385, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.3519, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0214, compute: 2.5558, data: 1.2412, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stdout>:2020-11-08 20:40:49,466 Mesh regression INFO: eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.4185, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0214, compute: 2.5558, data: 1.3442, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.4774, 2d joint loss: 0.0025, 3d joint loss: 0.0021, vertex loss: 0.0215, compute: 2.5558, data: 1.2434, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.3732, 2d joint loss: 0.0024, 3d joint loss: 0.0020, vertex loss: 0.0214, compute: 2.5558, data: 1.2390, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.3749, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0214, compute: 2.5558, data: 1.2388, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.3769, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0215, compute: 2.5558, data: 1.2399, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.4240, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0215, compute: 2.5558, data: 1.2401, lr: 0.000100
2020-11-08 20:40:51 [1,0]<stderr>:INFO:Mesh regression:eta: 1:45:38 epoch: 3 iter: 340 max mem : 27601  loss: 4.4185, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0214, compute: 2.5558, data: 1.3442, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.3869, 2d joint loss: 0.0025, 3d joint loss: 0.0020, vertex loss: 0.0212, compute: 2.4898, data: 1.1805, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.3020, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0212, compute: 2.4898, data: 1.1760, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.3623, 2d joint loss: 0.0025, 3d joint loss: 0.0020, vertex loss: 0.0211, compute: 2.4898, data: 1.1754, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stdout>:2020-11-08 20:41:16,834 Mesh regression INFO: eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.3260, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0211, compute: 2.4898, data: 1.2790, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.2601, 2d joint loss: 0.0022, 3d joint loss: 0.0019, vertex loss: 0.0211, compute: 2.4898, data: 1.1781, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.2828, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0211, compute: 2.4898, data: 1.1760, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.2837, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0212, compute: 2.4898, data: 1.1767, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:05 epoch: 3 iter: 360 max mem : 27601  loss: 4.3286, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0212, compute: 2.4898, data: 1.1769, lr: 0.000100
2020-11-08 20:41:18 [1,0]<stderr>:INFO:Mesh regression:eta: 1:42:04 epoch: 3 iter: 360 max mem : 27601  loss: 4.3260, 2d joint loss: 0.0023, 3d joint loss: 0.0020, vertex loss: 0.0211, compute: 2.4898, data: 1.2790, lr: 0.000100
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stdout>:2020-11-08 20:42:56,738 Mesh regression INFO: Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:42:57 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 4  mPVE:  81.31, mPJPE:  77.07, mPVE_smpl:  88.28, mPJPE_smpl:  77.10, PAmPJPE_smpl:  47.90, Data Count: 35520.00
2020-11-08 20:43:06 [1,0]<stdout>:2020-11-08 20:43:06,300 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-4-376
2020-11-08 20:43:06 [1,0]<stderr>:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-4-376
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.1738, 2d joint loss: 0.0022, 3d joint loss: 0.0019, vertex loss: 0.0208, compute: 2.6589, data: 1.3277, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2523, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0209, compute: 2.6589, data: 1.3255, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.3052, 2d joint loss: 0.0025, 3d joint loss: 0.0020, vertex loss: 0.0209, compute: 2.6590, data: 1.3296, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2847, 2d joint loss: 0.0024, 3d joint loss: 0.0020, vertex loss: 0.0209, compute: 2.6590, data: 1.3250, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2016, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0208, compute: 2.6590, data: 1.3255, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stdout>:2020-11-08 20:43:10,911 Mesh regression INFO: eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2446, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0208, compute: 2.6590, data: 1.4492, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2498, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0209, compute: 2.6590, data: 1.3265, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.1931, 2d joint loss: 0.0022, 3d joint loss: 0.0019, vertex loss: 0.0209, compute: 2.6590, data: 1.3262, lr: 0.000100
2020-11-08 20:43:12 [1,0]<stderr>:INFO:Mesh regression:eta: 1:48:07 epoch: 4 iter: 380 max mem : 27601  loss: 4.2446, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0208, compute: 2.6590, data: 1.4492, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stdout>:2020-11-08 20:43:37,518 Mesh regression INFO: eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.1662, 2d joint loss: 0.0022, 3d joint loss: 0.0019, vertex loss: 0.0206, compute: 2.5925, data: 1.3832, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.2039, 2d joint loss: 0.0024, 3d joint loss: 0.0019, vertex loss: 0.0206, compute: 2.5925, data: 1.2643, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.2139, 2d joint loss: 0.0024, 3d joint loss: 0.0019, vertex loss: 0.0207, compute: 2.5925, data: 1.2686, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.1186, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0206, compute: 2.5925, data: 1.2648, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.0937, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0206, compute: 2.5926, data: 1.2669, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.1625, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0206, compute: 2.5926, data: 1.2647, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.1779, 2d joint loss: 0.0023, 3d joint loss: 0.0019, vertex loss: 0.0207, compute: 2.5926, data: 1.2657, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:34 epoch: 4 iter: 400 max mem : 27601  loss: 4.1123, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0206, compute: 2.5926, data: 1.2655, lr: 0.000100
2020-11-08 20:43:39 [1,0]<stderr>:INFO:Mesh regression:eta: 1:44:33 epoch: 4 iter: 400 max mem : 27601  loss: 4.1662, 2d joint loss: 0.0022, 3d joint loss: 0.0019, vertex loss: 0.0206, compute: 2.5925, data: 1.3832, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.0161, 2d joint loss: 0.0021, 3d joint loss: 0.0018, vertex loss: 0.0203, compute: 2.5324, data: 1.2118, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.1555, 2d joint loss: 0.0024, 3d joint loss: 0.0019, vertex loss: 0.0204, compute: 2.5324, data: 1.2134, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.0946, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0204, compute: 2.5324, data: 1.2097, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.0946, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0204, compute: 2.5324, data: 1.2106, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.1248, 2d joint loss: 0.0024, 3d joint loss: 0.0019, vertex loss: 0.0204, compute: 2.5324, data: 1.2093, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.0421, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0203, compute: 2.5324, data: 1.2098, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stdout>:2020-11-08 20:44:04,125 Mesh regression INFO: eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.1027, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0204, compute: 2.5324, data: 1.3236, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.0706, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0204, compute: 2.5325, data: 1.2105, lr: 0.000100
2020-11-08 20:44:06 [1,0]<stderr>:INFO:Mesh regression:eta: 1:41:17 epoch: 4 iter: 420 max mem : 27601  loss: 4.1027, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0204, compute: 2.5324, data: 1.3236, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0353, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4774, data: 1.1598, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0748, 2d joint loss: 0.0024, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4774, data: 1.1632, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 3.9952, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4774, data: 1.1604, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 3.9579, 2d joint loss: 0.0021, 3d joint loss: 0.0017, vertex loss: 0.0201, compute: 2.4775, data: 1.1618, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stdout>:2020-11-08 20:44:30,589 Mesh regression INFO: eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0574, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4775, data: 1.2689, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0450, 2d joint loss: 0.0023, 3d joint loss: 0.0018, vertex loss: 0.0201, compute: 2.4775, data: 1.1592, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0177, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4775, data: 1.1606, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 3.9819, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0201, compute: 2.4775, data: 1.1598, lr: 0.000100
2020-11-08 20:44:33 [1,0]<stderr>:INFO:Mesh regression:eta: 1:38:16 epoch: 4 iter: 440 max mem : 27601  loss: 4.0574, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0202, compute: 2.4775, data: 1.2689, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.8918, 2d joint loss: 0.0021, 3d joint loss: 0.0017, vertex loss: 0.0199, compute: 2.4279, data: 1.1161, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9865, 2d joint loss: 0.0023, 3d joint loss: 0.0018, vertex loss: 0.0199, compute: 2.4279, data: 1.1136, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9110, 2d joint loss: 0.0022, 3d joint loss: 0.0017, vertex loss: 0.0199, compute: 2.4279, data: 1.1140, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stdout>:2020-11-08 20:44:57,356 Mesh regression INFO: eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9972, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0200, compute: 2.4279, data: 1.2197, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9705, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0200, compute: 2.4279, data: 1.1139, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9976, 2d joint loss: 0.0023, 3d joint loss: 0.0018, vertex loss: 0.0200, compute: 2.4279, data: 1.1174, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9512, 2d joint loss: 0.0022, 3d joint loss: 0.0017, vertex loss: 0.0200, compute: 2.4279, data: 1.1145, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:30 epoch: 4 iter: 460 max mem : 27601  loss: 3.9516, 2d joint loss: 0.0022, 3d joint loss: 0.0017, vertex loss: 0.0199, compute: 2.4280, data: 1.1147, lr: 0.000100
2020-11-08 20:44:57 [1,0]<stderr>:INFO:Mesh regression:eta: 1:35:29 epoch: 4 iter: 460 max mem : 27601  loss: 3.9972, 2d joint loss: 0.0022, 3d joint loss: 0.0018, vertex loss: 0.0200, compute: 2.4279, data: 1.2197, lr: 0.000100
2020-11-08 20:46:30 [1,0]<stdout>:2020-11-08 20:46:29,271 Mesh regression INFO: Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:30 [1,0]<stderr>:INFO:Mesh regression:Validation epoch: 5  mPVE:  82.48, mPJPE:  78.34, mPVE_smpl:  90.14, mPJPE_smpl:  78.34, PAmPJPE_smpl:  49.67, Data Count: 35520.00
2020-11-08 20:46:39 [1,0]<stdout>:2020-11-08 20:46:38,811 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-5-470
2020-11-08 20:46:39 [1,0]<stderr>:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-5-470
wjingdan commented 3 years ago

Thank you for your answer.

At 2021-09-25 06:38:51, "Kevin Ke-Yun Lin" @.***> wrote:

You may find our original training log for 3DPW as below.

Note: (1) At that time, we were trying to fine-tune for 30 epochs. But we found we already got good results at the early epoch (4th epoch as below). So, we terminated the training and didn't tune it further. (2) We have refactored the codebase before code release. The following log is from the old codebase. That's why the log style is a bit different.

2020-11-08 20:25:40 [1,0]:11-08 20:25:39.515 az-scus-v100-2-worker-mgfopa 573 aml_main.py:20 cmd_run(): start to cmd run: 2020-11-08 20:25:40 [1,0]:python -m torch.distributed.launch --nproc_per_node=8 tools/human_mesh/run_train_simplifiedmesh_2d3d_mvm_newbone_conv.py --train_yaml 3dpw_backup/train.yaml --val_yaml 3dpw_backup/test_has_gender.yaml --arch hrnet-w64 --model_name_or_path models/captioning/bert-base-uncased/ --num_workers 2 --logging_steps 20 --resume_checkpoint _output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin --per_gpu_train_batch_size 30 --per_gpu_eval_batch_size 30 --num_hidden_layers 4 --num_attention_heads 4 --lr 1e-4 --backbone_pretrained --fix_backbone 0 --object_query 1 --masking_inputs 0 --img_scale_factor 1 --scheduler iter_step --num_train_epochs 30 --input_feat_dim 2051,512,128 --hidden_feat_dim 1024,256,128 --vertices_loss_weight 100.0 --joints_loss_weight 1000.0 --vloss_w_full 0.33 --vloss_w_sub 0.33 --vloss_w_sub2 0.33 2020-11-08 20:25:40 [1,0]: 2020-11-08 20:25:40 [1,0]: 2020-11-08 20:25:40 [1,0]:Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 2020-11-08 20:25:40 [1,0]: 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 5 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 2 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 4 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 3 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 1 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 7 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 6 2020-11-08 20:25:43 [1,0]:Init distributed training on local rank 0 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:990 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:NCCL version 2.4.8+cuda10.1 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:995 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:996 [6] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:996 [6] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:995 [5] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:992 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:991 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:992 [2] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:991 [1] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:993 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:993 [3] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:994 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:994 [4] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO Bootstrap : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). 2020-11-08 20:25:49 [1,0]: 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:997 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1] 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:997 [7] NCCL INFO NET/Socket : Using [0]ib0:172.16.1.10<0> 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO NCCL_TREE_THRESHOLD set by environment to 0. 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 00 : 0 1 2 3 7 4 6 5 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 01 : 0 2 4 5 7 6 1 3 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 02 : 0 3 1 6 7 5 4 2 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 03 : 0 3 2 1 6 4 7 5 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 04 : 0 5 6 4 7 3 2 1 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 05 : 0 5 7 4 6 1 2 3 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 06 : 0 1 2 3 7 4 6 5 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 07 : 0 2 4 5 7 6 1 3 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 08 : 0 3 1 6 7 5 4 2 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 09 : 0 3 2 1 6 4 7 5 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 10 : 0 5 6 4 7 3 2 1 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Channel 11 : 0 5 7 4 6 1 2 3 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 00 : 1[2] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 00 : 3[7] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 00 : 4[0] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 00 : 2[1] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 00 : 7[6] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 00 : 5[4] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 00 : 6[5] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 00 : 0[3] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 01 : 1[2] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 01 : 3[7] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 01 : 4[0] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 01 : 2[1] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 01 : 7[6] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 01 : 5[4] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 01 : 6[5] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 01 : 0[3] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 02 : 1[2] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 02 : 3[7] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 02 : 4[0] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 02 : 7[6] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 02 : 2[1] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 02 : 0[3] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 02 : 6[5] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 02 : 5[4] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 03 : 1[2] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 03 : 3[7] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 03 : 4[0] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 03 : 7[6] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 03 : 2[1] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 03 : 0[3] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 03 : 5[4] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 03 : 6[5] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 04 : 3[7] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 04 : 1[2] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 04 : 4[0] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 04 : 7[6] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 04 : 2[1] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 04 : 0[3] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 04 : 5[4] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 04 : 6[5] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 05 : 1[2] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 05 : 3[7] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 05 : 4[0] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 05 : 7[6] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 05 : 2[1] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 05 : 0[3] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 05 : 6[5] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 05 : 5[4] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 06 : 3[7] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 06 : 4[0] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 06 : 2[1] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 06 : 1[2] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 06 : 7[6] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 06 : 0[3] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 06 : 6[5] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 06 : 5[4] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 07 : 3[7] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 07 : 4[0] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 07 : 2[1] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 07 : 1[2] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 07 : 7[6] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 07 : 0[3] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 07 : 6[5] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 07 : 5[4] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 08 : 3[7] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 08 : 4[0] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 08 : 2[1] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 08 : 1[2] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 08 : 7[6] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 08 : 0[3] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 08 : 6[5] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 08 : 5[4] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 09 : 3[7] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 09 : 2[1] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 09 : 4[0] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 09 : 1[2] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 09 : 7[6] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 09 : 0[3] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 09 : 6[5] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 09 : 5[4] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 10 : 3[7] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 10 : 2[1] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 10 : 4[0] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 10 : 1[2] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 10 : 7[6] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 10 : 0[3] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 10 : 6[5] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 10 : 5[4] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO Ring 11 : 3[7] -> 0[3] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO Ring 11 : 1[2] -> 2[1] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO Ring 11 : 4[0] -> 6[5] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO Ring 11 : 7[6] -> 4[0] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO Ring 11 : 2[1] -> 3[7] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO Ring 11 : 6[5] -> 1[2] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Ring 11 : 0[3] -> 5[4] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO Ring 11 : 5[4] -> 7[6] via P2P/IPC 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:993:1106 [3] NCCL INFO comm 0x7f73e4001d50 rank 3 nranks 8 cudaDev 3 nvmlDev 7 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:994:1107 [4] NCCL INFO comm 0x7f8cd0001d50 rank 4 nranks 8 cudaDev 4 nvmlDev 0 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:997:1108 [7] NCCL INFO comm 0x7f43d8001d50 rank 7 nranks 8 cudaDev 7 nvmlDev 6 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:991:1105 [1] NCCL INFO comm 0x7f47d8001d50 rank 1 nranks 8 cudaDev 1 nvmlDev 2 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:992:1104 [2] NCCL INFO comm 0x7f64f4001d50 rank 2 nranks 8 cudaDev 2 nvmlDev 1 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:996:1102 [6] NCCL INFO comm 0x7f7d08001d50 rank 6 nranks 8 cudaDev 6 nvmlDev 5 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:1101 [0] NCCL INFO comm 0x7f79e8001d50 rank 0 nranks 8 cudaDev 0 nvmlDev 3 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:995:1103 [5] NCCL INFO comm 0x7fb7a8001d50 rank 5 nranks 8 cudaDev 5 nvmlDev 4 - Init COMPLETE 2020-11-08 20:25:49 [1,0]:az-scus-v100-2-worker-mgfopa:990:990 [0] NCCL INFO Launch mode Parallel 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:49 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:49 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:52 [1,0]:2020-11-08 20:25:50,871 Mesh regression INFO: Using 8 GPUs 2020-11-08 20:25:52 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:52 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:55 [1,0]:/miniconda/envs/py37/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient. 2020-11-08 20:25:55 [1,0]: self._set_intXint(row, col, x.flat[0]) 2020-11-08 20:25:58 [1,0]:2020-11-08 20:25:57,499 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4 2020-11-08 20:25:58 [1,0]:2020-11-08 20:25:57,759 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,410 Mesh regression INFO: Init model from scratch. 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,459 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,509 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,849 Mesh regression INFO: Init model from scratch. 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,898 Mesh regression INFO: Update config parameter num_hidden_layers: 12 -> 4 2020-11-08 20:26:01 [1,0]:2020-11-08 20:25:59,946 Mesh regression INFO: Update config parameter num_attention_heads: 12 -> 4 2020-11-08 20:26:01 [1,0]:2020-11-08 20:26:00,139 Mesh regression INFO: Init model from scratch. 2020-11-08 20:26:08 [1,0]:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth 2020-11-08 20:26:08 [1,0]:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth 2020-11-08 20:26:08 [1,0]: 2020-11-08 20:26:08 [1,0]: 2020-11-08 20:26:08 [1,0]:=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth 2020-11-08 20:26:08 [1,0]:2020-11-08 20:26:07,957 Mesh regression INFO: => loading hrnet-v2-w64 model 2020-11-08 20:26:08 [1,0]:2020-11-08 20:26:08,007 Mesh regression INFO: Transformer Encoder 2 total parameters: 102256646 2020-11-08 20:26:08 [1,0]:2020-11-08 20:26:08,072 Mesh regression INFO: Backbone model total parameters: 128059944 2020-11-08 20:26:08 [1,0]:2020-11-08 20:26:08,189 Mesh regression INFO: Loading state dict from checkpoint _output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:2020-11-08 20:26:18,607 Mesh regression INFO: Training parameters Namespace(adam_epsilon=1e-08, arch='hrnet-w64', backbone_pretrained=True, config_name='', data_dir='datasets', device=device(type='cuda'), distributed=True, do_lower_case=False, drop_out=0.1, effective_batch_size=-1, fix_backbone=0, hidden_feat_dim='1024,256,128', hidden_size=-1, img_feature_dim=2051, img_scale_factor=1, input_feat_dim='2051,512,128', intermediate_size=-1, joints_loss_weight=1000.0, load_partial_weights=False, local_rank=0, logging_steps=20, lr=0.0001, mask_prob=0.15, mask_type='bidirectional', masking_inputs=0, max_masked_tokens=3, model_name_or_path='models/captioning/bert-base-uncased/', momentum=0.9, no_sort_by_conf=False, num_attention_heads=4, num_gpus=8, num_hidden_layers=4, num_train_epochs=30, num_workers=2, object_query=1, on_memory=False, output_dir='_keli/output/', per_gpu_eval_batch_size=30, per_gpu_train_batch_size=30, resume_checkpoint='_output/20201103_Mesh2d3d_MVM_CovImgToken_Tax-H36m-coco40k-Muco-UP-Mpii_arch.hrnet-w64.bert-L6_bs.30_hidl.4_head.4_lr.1e-4_ep.200_vloss.100.0_jloss.1000.0_isz.2051,512,128_hsz.1024,256,128_jregloss_multiresl_full.0.33_sub.0.33_sub2.0.33_LearnAllUp_2Djloss100/checkpoint-96-190368/model.bin', run_eval_only=False, save_steps=50000, scheduler='iter_step', seed=88, tokenizer_name='', train_yaml='3dpw_backup/train.yaml', val_yaml='3dpw_backup/test_has_gender.yaml', val_yaml2='imagenet2012/test.yaml', val_yaml3='imagenet2012/test.yaml', vertices_loss_weight=100.0, vloss_w_full=0.33, vloss_w_sub=0.33, vloss_w_sub2=0.33, warmup_steps=0, weight_decay=0.05) 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/train.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/test_has_gender.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/test_has_gender.yaml 2020-11-08 20:26:20 [1,0]:3dpw_backup/test_has_gender.yaml 2020-11-08 20:22020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 10.9480, 2d joint loss: 0.0066, 3d joint loss: 0.0066, vertex loss: 0.0373, compute: 4.2248, data: 2.8355, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:08 epoch: 0 iter: 20 max mem : 27601 loss: 11.1114, 2d joint loss: 0.0063, 3d joint loss: 0.0067, vertex loss: 0.0382, compute: 4.2246, data: 2.8432, lr: 0.000100 2020-11-08 20:27:47 [1,0]:2020-11-08 20:27:44,999 Mesh regression INFO: eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 12.2818, 2d joint loss: 0.0067, 3d joint loss: 0.0076, vertex loss: 0.0396, compute: 4.2247, data: 2.8407, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 10.8245, 2d joint loss: 0.0062, 3d joint loss: 0.0064, vertex loss: 0.0380, compute: 4.2249, data: 2.8416, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 11.5213, 2d joint loss: 0.0067, 3d joint loss: 0.0070, vertex loss: 0.0390, compute: 4.2248, data: 2.8314, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 11.7696, 2d joint loss: 0.0065, 3d joint loss: 0.0072, vertex loss: 0.0392, compute: 4.2249, data: 2.8450, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:10 epoch: 0 iter: 20 max mem : 27601 loss: 11.1805, 2d joint loss: 0.0062, 3d joint loss: 0.0067, vertex loss: 0.0389, compute: 4.2253, data: 2.8389, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:11 epoch: 0 iter: 20 max mem : 27601 loss: 11.0279, 2d joint loss: 0.0059, 3d joint loss: 0.0066, vertex loss: 0.0388, compute: 4.2254, data: 2.8384, lr: 0.000100 2020-11-08 20:27:47 [1,0]:INFO:Mesh regression:eta: 3:17:09 epoch: 0 iter: 20 max mem : 27601 loss: 12.2818, 2d joint loss: 0.0067, 3d joint loss: 0.0076, vertex loss: 0.0396, compute: 4.2247, data: 2.8407, lr: 0.000100 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:240: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:27:47 [1,0]:/tmp/code/maskrcnn_benchmark/modeling/human_mesh/smpl_model/renderer.py:72: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. 2020-11-08 20:27:47 [1,0]: if np.issubdtype(image.dtype, np.float): 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:11 epoch: 0 iter: 40 max mem : 27601 loss: 9.1141, 2d joint loss: 0.0048, 3d joint loss: 0.0052, vertex loss: 0.0342, compute: 2.8099, data: 1.4785, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.0047, 2d joint loss: 0.0052, 3d joint loss: 0.0052, vertex loss: 0.0332, compute: 2.8102, data: 1.4750, lr: 0.000100 2020-11-08 20:28:14 [1,0]:2020-11-08 20:28:12,912 Mesh regression INFO: eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.3616, 2d joint loss: 0.0048, 3d joint loss: 0.0055, vertex loss: 0.0341, compute: 2.8102, data: 1.5135, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.2867, 2d joint loss: 0.0052, 3d joint loss: 0.0053, vertex loss: 0.0343, compute: 2.8103, data: 1.4741, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.4480, 2d joint loss: 0.0053, 3d joint loss: 0.0055, vertex loss: 0.0345, compute: 2.8103, data: 1.4804, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.3447, 2d joint loss: 0.0051, 3d joint loss: 0.0054, vertex loss: 0.0343, compute: 2.8103, data: 1.4807, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:13 epoch: 0 iter: 40 max mem : 27601 loss: 8.8068, 2d joint loss: 0.0047, 3d joint loss: 0.0050, vertex loss: 0.0338, compute: 2.8104, data: 1.4793, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:13 epoch: 0 iter: 40 max mem : 27601 loss: 8.7020, 2d joint loss: 0.0046, 3d joint loss: 0.0049, vertex loss: 0.0336, compute: 2.8105, data: 1.4803, lr: 0.000100 2020-11-08 20:28:14 [1,0]:INFO:Mesh regression:eta: 2:10:12 epoch: 0 iter: 40 max mem : 27601 loss: 9.3616, 2d joint loss: 0.0048, 3d joint loss: 0.0055, vertex loss: 0.0341, compute: 2.8102, data: 1.5135, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 7.8472, 2d joint loss: 0.0044, 3d joint loss: 0.0043, vertex loss: 0.0307, compute: 2.3186, data: 1.0205, lr: 0.000100 2020-11-08 20:28:41 [1,0]:2020-11-08 20:28:39,616 Mesh regression INFO: eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 8.2019, 2d joint loss: 0.0041, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3185, data: 1.0535, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 7.9426, 2d joint loss: 0.0041, 3d joint loss: 0.0044, vertex loss: 0.0314, compute: 2.3186, data: 1.0238, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 8.0331, 2d joint loss: 0.0043, 3d joint loss: 0.0045, vertex loss: 0.0315, compute: 2.3186, data: 1.0201, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 7.5853, 2d joint loss: 0.0040, 3d joint loss: 0.0041, vertex loss: 0.0309, compute: 2.3186, data: 1.0239, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 8.2093, 2d joint loss: 0.0043, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3187, data: 1.0245, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 8.2107, 2d joint loss: 0.0046, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3187, data: 1.0244, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 7.8058, 2d joint loss: 0.0041, 3d joint loss: 0.0043, vertex loss: 0.0313, compute: 2.3188, data: 1.0236, lr: 0.000100 2020-11-08 20:28:41 [1,0]:INFO:Mesh regression:eta: 1:46:39 epoch: 0 iter: 60 max mem : 27601 loss: 8.2019, 2d joint loss: 0.0041, 3d joint loss: 0.0046, vertex loss: 0.0316, compute: 2.3185, data: 1.0535, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.1706, 2d joint loss: 0.0040, 3d joint loss: 0.0039, vertex loss: 0.0291, compute: 2.0746, data: 0.7931, lr: 0.000100 2020-11-08 20:29:08 [1,0]:2020-11-08 20:29:06,469 Mesh regression INFO: eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.4384, 2d joint loss: 0.0037, 3d joint loss: 0.0041, vertex loss: 0.0298, compute: 2.0745, data: 0.8243, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.2779, 2d joint loss: 0.0037, 3d joint loss: 0.0039, vertex loss: 0.0298, compute: 2.0746, data: 0.7959, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.3775, 2d joint loss: 0.0038, 3d joint loss: 0.0040, vertex loss: 0.0296, compute: 2.0746, data: 0.7962, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.3226, 2d joint loss: 0.0039, 3d joint loss: 0.0040, vertex loss: 0.0297, compute: 2.0746, data: 0.7929, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 6.9406, 2d joint loss: 0.0036, 3d joint loss: 0.0037, vertex loss: 0.0292, compute: 2.0747, data: 0.7958, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.1073, 2d joint loss: 0.0037, 3d joint loss: 0.0038, vertex loss: 0.0294, compute: 2.0748, data: 0.7959, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:45 epoch: 0 iter: 80 max mem : 27601 loss: 7.6323, 2d joint loss: 0.0043, 3d joint loss: 0.0042, vertex loss: 0.0299, compute: 2.0750, data: 0.7958, lr: 0.000100 2020-11-08 20:29:08 [1,0]:INFO:Mesh regression:eta: 1:34:44 epoch: 0 iter: 80 max mem : 27601 loss: 7.4384, 2d joint loss: 0.0037, 3d joint loss: 0.0041, vertex loss: 0.0298, compute: 2.0745, data: 0.8243, lr: 0.000100 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:2020-11-08 20:32:19,937 Mesh regression INFO: Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:20 [1,0]:INFO:Mesh regression:Validation epoch: 1 mPVE: 80.65, mPJPE: 76.19, mPVE_smpl: 90.18, mPJPE_smpl: 76.88, PAmPJPE_smpl: 48.04, Data Count: 35520.00 2020-11-08 20:32:26 [1,0]:2020-11-08 20:32:26,475 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-1-94 2020-11-08 20:32:26 [1,0]:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-1-94 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.7488, 2d joint loss: 0.0034, 3d joint loss: 0.0036, vertex loss: 0.0283, compute: 3.7296, data: 2.4009, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 7.0071, 2d joint loss: 0.0040, 3d joint loss: 0.0038, vertex loss: 0.0284, compute: 3.7296, data: 2.4002, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.7134, 2d joint loss: 0.0037, 3d joint loss: 0.0035, vertex loss: 0.0280, compute: 3.7297, data: 2.3979, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.6355, 2d joint loss: 0.0034, 3d joint loss: 0.0035, vertex loss: 0.0281, compute: 3.7297, data: 2.3990, lr: 0.000100 2020-11-08 20:32:35 [1,0]:2020-11-08 20:32:33,473 Mesh regression INFO: eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.8787, 2d joint loss: 0.0034, 3d joint loss: 0.0037, vertex loss: 0.0284, compute: 3.7297, data: 2.4923, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.6848, 2d joint loss: 0.0035, 3d joint loss: 0.0035, vertex loss: 0.0281, compute: 3.7297, data: 2.3968, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.5730, 2d joint loss: 0.0034, 3d joint loss: 0.0034, vertex loss: 0.0282, compute: 3.7297, data: 2.4002, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.9428, 2d joint loss: 0.0036, 3d joint loss: 0.0037, vertex loss: 0.0285, compute: 3.7298, data: 2.3993, lr: 0.000100 2020-11-08 20:32:35 [1,0]:INFO:Mesh regression:eta: 2:49:04 epoch: 1 iter: 100 max mem : 27601 loss: 6.8787, 2d joint loss: 0.0034, 3d joint loss: 0.0037, vertex loss: 0.0284, compute: 3.7297, data: 2.4923, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.2608, 2d joint loss: 0.0034, 3d joint loss: 0.0032, vertex loss: 0.0268, compute: 3.3308, data: 2.0158, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.5182, 2d joint loss: 0.0037, 3d joint loss: 0.0034, vertex loss: 0.0272, compute: 3.3308, data: 2.0188, lr: 0.000100 2020-11-08 20:33:02 [1,0]:2020-11-08 20:33:00,197 Mesh regression INFO: eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.3791, 2d joint loss: 0.0031, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3308, data: 2.0993, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.2357, 2d joint loss: 0.0032, 3d joint loss: 0.0032, vertex loss: 0.0270, compute: 3.3308, data: 2.0167, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.4800, 2d joint loss: 0.0033, 3d joint loss: 0.0034, vertex loss: 0.0274, compute: 3.3308, data: 2.0169, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.1705, 2d joint loss: 0.0032, 3d joint loss: 0.0031, vertex loss: 0.0271, compute: 3.3308, data: 2.0192, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.2436, 2d joint loss: 0.0032, 3d joint loss: 0.0032, vertex loss: 0.0270, compute: 3.3308, data: 2.0159, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.2898, 2d joint loss: 0.0032, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3309, data: 2.0196, lr: 0.000100 2020-11-08 20:33:02 [1,0]:INFO:Mesh regression:eta: 2:29:53 epoch: 1 iter: 120 max mem : 27601 loss: 6.3791, 2d joint loss: 0.0031, 3d joint loss: 0.0033, vertex loss: 0.0271, compute: 3.3308, data: 2.0993, lr: 0.000100 2020-11-08 20:33:29 [1,0]:2020-11-08 20:33:27,007 Mesh regression INFO: eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 6.0413, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0262, compute: 3.0464, data: 1.8175, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 6.0003, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0263, compute: 3.0465, data: 1.7471, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 6.1838, 2d joint loss: 0.0035, 3d joint loss: 0.0032, vertex loss: 0.0264, compute: 3.0465, data: 1.7499, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 5.9973, 2d joint loss: 0.0033, 3d joint loss: 0.0031, vertex loss: 0.0260, compute: 3.0465, data: 1.7436, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 5.9437, 2d joint loss: 0.0031, 3d joint loss: 0.0030, vertex loss: 0.0261, compute: 3.0465, data: 1.7455, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 5.8686, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0262, compute: 3.0465, data: 1.7466, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 5.9656, 2d joint loss: 0.0031, 3d joint loss: 0.0030, vertex loss: 0.0262, compute: 3.0465, data: 1.7437, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 6.0839, 2d joint loss: 0.0031, 3d joint loss: 0.0031, vertex loss: 0.0263, compute: 3.0465, data: 1.7471, lr: 0.000100 2020-11-08 20:33:29 [1,0]:INFO:Mesh regression:eta: 2:16:04 epoch: 1 iter: 140 max mem : 27601 loss: 6.0413, 2d joint loss: 0.0030, 3d joint loss: 0.0031, vertex loss: 0.0262, compute: 3.0464, data: 1.8175, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.8663, 2d joint loss: 0.0033, 3d joint loss: 0.0030, vertex loss: 0.0256, compute: 2.8316, data: 1.5449, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.7580, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0256, compute: 2.8316, data: 1.5426, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.6847, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.5394, lr: 0.000100 2020-11-08 20:33:53 [1,0]:2020-11-08 20:33:53,572 Mesh regression INFO: eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.7673, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.6062, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.6910, 2d joint loss: 0.0031, 3d joint loss: 0.0029, vertex loss: 0.0252, compute: 2.8317, data: 1.5392, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.7129, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.5409, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.6514, 2d joint loss: 0.0029, 3d joint loss: 0.0028, vertex loss: 0.0255, compute: 2.8317, data: 1.5419, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.7965, 2d joint loss: 0.0030, 3d joint loss: 0.0029, vertex loss: 0.0255, compute: 2.8318, data: 1.5424, lr: 0.000100 2020-11-08 20:33:53 [1,0]:INFO:Mesh regression:eta: 2:05:32 epoch: 1 iter: 160 max mem : 27601 loss: 5.7673, 2d joint loss: 0.0029, 3d joint loss: 0.0029, vertex loss: 0.0254, compute: 2.8317, data: 1.6062, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601 loss: 5.4878, 2d joint loss: 0.0029, 3d joint loss: 0.0027, vertex loss: 0.0248, compute: 2.6734, data: 1.3818, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601 loss: 5.6730, 2d joint loss: 0.0032, 3d joint loss: 0.0029, vertex loss: 0.0249, compute: 2.6734, data: 1.3874, lr: 0.000100 2020-11-08 20:34:23 [1,0]:2020-11-08 20:34:21,727 Mesh regression INFO: eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601 loss: 5.5217, 2d joint loss: 0.0027, 3d joint loss: 0.0028, vertex loss: 0.0247, compute: 2.6734, data: 1.4504, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601 loss: 5.4529, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0247, compute: 2.6735, data: 1.3806, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601 loss: 5.4541, 2d joint loss: 0.0029, 3d joint loss: 0.0027, vertex loss: 0.0246, compute: 2.6735, data: 1.3804, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601 loss: 5.5094, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0249, compute: 2.6735, data: 1.3835, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601 loss: 5.5450, 2d joint loss: 0.0028, 3d joint loss: 0.0028, vertex loss: 0.0249, compute: 2.6735, data: 1.3832, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:38 epoch: 1 iter: 180 max mem : 27601 loss: 5.4613, 2d joint loss: 0.0028, 3d joint loss: 0.0027, vertex loss: 0.0249, compute: 2.6736, data: 1.3828, lr: 0.000100 2020-11-08 20:34:23 [1,0]:INFO:Mesh regression:eta: 1:57:37 epoch: 1 iter: 180 max mem : 27601 loss: 5.5217, 2d joint loss: 0.0027, 3d joint loss: 0.0028, vertex loss: 0.0247, compute: 2.6734, data: 1.4504, lr: 0.000100 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:2020-11-08 20:35:51,460 Mesh regression INFO: Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:35:54 [1,0]:INFO:Mesh regression:Validation epoch: 2 mPVE: 81.17, mPJPE: 76.61, mPVE_smpl: 90.06, mPJPE_smpl: 77.41, PAmPJPE_smpl: 48.42, Data Count: 35520.00 2020-11-08 20:36:03 [1,0]:2020-11-08 20:36:00,965 Mesh regression INFO: Save checkpoint to _keli/output/checkpoint-2-188 2020-11-08 20:36:03 [1,0]:INFO:Mesh regression:Save checkpoint to _keli/output/checkpoint-2-188 2020-11-08 20:36:18 [1,0]:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601 loss: 5.3231, 2d joint loss: 0.0027, 3d joint loss: 0.0026, vertex loss: 0.0243, compute: 2.9736, data: 1.6445, lr: 0.000100 2020-11-08 20:36:18 [1,0]:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601 loss: 5.2614, 2d joint loss: 0.0029, 3d joint loss: 0.0026, vertex loss: 0.0240, compute: 2.9737, data: 1.6414, lr: 0.000100 2020-11-08 20:36:18 [1,0]:INFO:Mesh regression:eta: 2:09:50 epoch: 2 iter: 200 max mem : 27601 loss: 5.4484, 2d joint loss: 0.0030, 3d joint loss: 0.0027, vertex loss: 0.0244, compute: 2.9736, data: 1.6495, lr: 0.000100 2020-11-08 20:36:18 [1,0]:INFO:Mesh regression:eta: 2:09:51 epoch: 2 iter: 200 max mem : 27601 loss: 5.2795, 2d joint loss: 0.0028, 3d joint loss: 0.0026, vertex loss: 0.0242, compute: 2.9737, data: 1.6426, lr: 0.000100 2020-11-08 20:36:18 [1,0]

wjingdan commented 3 years ago

Thank you for your answer.