search

DirtyHarryLYL / DJ-RN

As a part of HAKE project (HAKE-3D). Code for our CVPR2020 paper "Detailed 2D-3D Joint Representation for Human-Object Interaction".
Apache License 2.0
100 stars 13 forks source link

NaN loss value, stopping! --> File "/home/mona/research/code/smplify-x/smplifyx/fit_single_frame.py", line 366, in fit_single_frame tqdm.write('Camera initialization final loss {:.4f}'.format( TypeError: unsupported format string passed to NoneType.__format__ #28

Closed monacv closed 3 years ago

monacv commented 3 years ago

After processing some images successfully, I got this error. Is there a fix for it?

Processing: ../../data/smplify-x/djrn_test_data/images/HICO_test2015_00000469.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 0.7507
Camera initialization final loss 1283.9620
Stage 000 done after 2.0166 seconds                                                                                                
Stage 001 done after 1.8066 seconds                                                                                                
Stage 002 done after 1.7855 seconds                                                                                                
Stage 003 done after 6.2607 seconds                                                                                                
Stage 004 done after 6.4354 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:18<00:00,  3.66s/it]
Body fitting Orientation 0 done after 18.3112 seconds                                                                              
Body final loss val = 7074.32373                                                                                                   
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:18<00:00, 18.31s/it]
Processing: ../../data/smplify-x/djrn_test_data/images/HICO_test2015_00000470.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 0.9054
Camera initialization final loss 805.9324
Stage 000 done after 2.0088 seconds                                                                                                
Stage 001 done after 0.6831 seconds                                                                                                
Stage 002 done after 2.9054 seconds                                                                                                
Stage 003 done after 7.4240 seconds                                                                                                
Stage 004 done after 1.3566 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:14<00:00,  2.88s/it]
Body fitting Orientation 0 done after 14.3841 seconds                                                                              
Body final loss val = 1076.25085                                                                                                   
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:14<00:00, 14.38s/it]
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 0.8964
Camera initialization final loss 10146.6113
Stage 000 done after 2.3258 seconds                                                                                                
Stage 001 done after 1.5916 seconds                                                                                                
Stage 002 done after 3.7246 seconds                                                                                                
Stage 003 done after 13.3136 seconds                                                                                               
Stage 004 done after 3.2600 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:24<00:00,  4.84s/it]
Body fitting Orientation 0 done after 24.2220 seconds                                                                              
Body final loss val = 196141.98438                                                                                                 
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.22s/it]
Processing: ../../data/smplify-x/djrn_test_data/images/HICO_test2015_00000471.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
NaN loss value, stopping!
Camera initialization done after 1.2850
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 245, in main
    fit_single_frame(img, keypoints[[person_id]],
  File "/home/mona/research/code/smplify-x/smplifyx/fit_single_frame.py", line 366, in fit_single_frame
    tqdm.write('Camera initialization final loss {:.4f}'.format(
TypeError: unsupported format string passed to NoneType.__format__

I am running it as follows:

$ cat djrn_fit.sh 
export CUDA_VISIBLE_DEVICES=0
python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder ../../data/smplify-x/djrn_train_data/ --output_folder ../../data/smplify-x/djrn_train_results --visualize="False" --model_folder ../../data/smplify-x/models_smplx_v1_1/models/smplx/SMPLX_NEUTRAL.npz --vposer_ckpt ../../data/smplify-x/vposer_v1_0 --part_segm_fn ../../data/smplify-x/smplx_parts_segm.pkl

python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder ../../data/smplify-x/djrn_test_data/ --output_folder ../../data/smplify-x/djrn_test_results --visualize="False" --model_folder ../../data/smplify-x/models_smplx_v1_1/models/smplx/SMPLX_NEUTRAL.npz --vposer_ckpt ../../data/smplify-x/vposer_v1_0 --part_segm_fn ../../data/smplify-x/smplx_parts_segm.pkl

I have:

~/research/data/smplify-x/djrn_train_results/meshes
$ numdir
293
~/research/data/smplify-x/djrn_test_results/meshes
$ numdir
338

Where numdir is: alias numdir='echofind . -type d | wc -l-1 | bc'

monacv commented 3 years ago

I also ran it on the server and only for train data and get the same exact error but for a different image:

Processing: ../../data/smplify-x/djrn_train_data/images/HICO_train2015_00000419.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 1.7629
Camera initialization final loss 3377.0957
Stage 000 done after 4.0044 seconds
Stage 001 done after 3.5054 seconds
Stage 002 done after 7.3627 seconds
Stage 003 done after 12.8516 seconds
Orientation:   0%|                                                                                           | 0/1 [00:27<?, ?it/sNaN loss value, stopping!███████████████████████████████████████████████████████████▏                 | 4/5 [00:27<00:08,  8.33s/it]
Stage 004 done after 5.7050 seconds
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:33<00:00,  6.69s/it]
Body fitting Orientation 0 done after 33.4430 seconds
Body final loss val = 76030.69531
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:33<00:00, 33.44s/it]
Processing: ../../data/smplify-x/djrn_train_data/images/HICO_train2015_00000420.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
NaN loss value, stopping!
Camera initialization done after 2.5138
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 262, in main
    **args)
  File "~/code/smplify-x/smplifyx/fit_single_frame.py", line 367, in fit_single_frame
    cam_init_loss_val))
TypeError: unsupported format string passed to NoneType.__format__

The process has been ongoing on the server for test images and currently is at 

~/data/smplify-x/djrn_test_results/meshes$ numdir
869

while

~/data/smplify-x/djrn_train_results/meshes$ numdir
293

This is the list of images processed on my local machine for train set of HICO:

~/research/data/smplify-x/djrn_test_results/meshes
$ ls 
total 1.4M
drwxrwxr-x   5 mona mona 4.0K Jan 11 15:54 ..
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:23 HICO_test2015_00000002
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:23 HICO_test2015_00000003
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:24 HICO_test2015_00000004
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:25 HICO_test2015_00000006
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:26 HICO_test2015_00000007
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:27 HICO_test2015_00000009
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:28 HICO_test2015_00000010
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:29 HICO_test2015_00000011
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:30 HICO_test2015_00000012
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:31 HICO_test2015_00000013
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:31 HICO_test2015_00000014
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:32 HICO_test2015_00000015
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:33 HICO_test2015_00000016
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:34 HICO_test2015_00000019
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:36 HICO_test2015_00000021
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:37 HICO_test2015_00000022
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:39 HICO_test2015_00000023
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:40 HICO_test2015_00000024
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:41 HICO_test2015_00000026
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:42 HICO_test2015_00000028
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:43 HICO_test2015_00000029
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:45 HICO_test2015_00000030
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:45 HICO_test2015_00000031
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:46 HICO_test2015_00000033
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:46 HICO_test2015_00000034
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:47 HICO_test2015_00000035
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:48 HICO_test2015_00000037
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:49 HICO_test2015_00000038
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:50 HICO_test2015_00000040
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:51 HICO_test2015_00000041
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:52 HICO_test2015_00000043
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:53 HICO_test2015_00000044
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:54 HICO_test2015_00000045
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:54 HICO_test2015_00000046
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:55 HICO_test2015_00000047
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:57 HICO_test2015_00000048
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:58 HICO_test2015_00000050
drwxrwxr-x   2 mona mona 4.0K Jan 11 22:59 HICO_test2015_00000051
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:01 HICO_test2015_00000052
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:02 HICO_test2015_00000053
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:02 HICO_test2015_00000057
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:04 HICO_test2015_00000058
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:04 HICO_test2015_00000059
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:05 HICO_test2015_00000060
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:05 HICO_test2015_00000061
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:06 HICO_test2015_00000063
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:07 HICO_test2015_00000065
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:07 HICO_test2015_00000068
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:08 HICO_test2015_00000070
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:09 HICO_test2015_00000071
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:10 HICO_test2015_00000072
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:11 HICO_test2015_00000073
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:12 HICO_test2015_00000074
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:12 HICO_test2015_00000077
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:14 HICO_test2015_00000078
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:15 HICO_test2015_00000079
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:16 HICO_test2015_00000080
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:16 HICO_test2015_00000082
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:17 HICO_test2015_00000083
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:18 HICO_test2015_00000084
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:18 HICO_test2015_00000086
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:20 HICO_test2015_00000088
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:21 HICO_test2015_00000090
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:22 HICO_test2015_00000091
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:23 HICO_test2015_00000092
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:23 HICO_test2015_00000093
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:24 HICO_test2015_00000096
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:25 HICO_test2015_00000097
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:27 HICO_test2015_00000098
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:28 HICO_test2015_00000099
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:28 HICO_test2015_00000100
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:29 HICO_test2015_00000102
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:31 HICO_test2015_00000103
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:31 HICO_test2015_00000106
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:32 HICO_test2015_00000107
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:33 HICO_test2015_00000108
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:33 HICO_test2015_00000109
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:34 HICO_test2015_00000110
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:35 HICO_test2015_00000112
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:36 HICO_test2015_00000113
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:38 HICO_test2015_00000114
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:39 HICO_test2015_00000115
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:40 HICO_test2015_00000116
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:42 HICO_test2015_00000117
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:42 HICO_test2015_00000118
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:43 HICO_test2015_00000120
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:44 HICO_test2015_00000121
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:45 HICO_test2015_00000123
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:47 HICO_test2015_00000125
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:49 HICO_test2015_00000126
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:49 HICO_test2015_00000127
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:50 HICO_test2015_00000128
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:50 HICO_test2015_00000129
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:51 HICO_test2015_00000130
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:52 HICO_test2015_00000131
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:53 HICO_test2015_00000132
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:54 HICO_test2015_00000133
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:54 HICO_test2015_00000134
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:55 HICO_test2015_00000135
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:56 HICO_test2015_00000136
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:57 HICO_test2015_00000137
drwxrwxr-x   2 mona mona 4.0K Jan 11 23:58 HICO_test2015_00000138
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:00 HICO_test2015_00000140
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:00 HICO_test2015_00000142
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:01 HICO_test2015_00000143
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:02 HICO_test2015_00000144
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:03 HICO_test2015_00000145
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:04 HICO_test2015_00000146
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:05 HICO_test2015_00000147
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:06 HICO_test2015_00000149
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:06 HICO_test2015_00000150
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:09 HICO_test2015_00000152
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:09 HICO_test2015_00000153
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:09 HICO_test2015_00000155
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:11 HICO_test2015_00000157
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:13 HICO_test2015_00000158
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:14 HICO_test2015_00000159
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:14 HICO_test2015_00000160
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:15 HICO_test2015_00000163
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:16 HICO_test2015_00000166
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:17 HICO_test2015_00000167
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:18 HICO_test2015_00000168
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:19 HICO_test2015_00000169
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:20 HICO_test2015_00000170
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:21 HICO_test2015_00000171
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:22 HICO_test2015_00000176
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:23 HICO_test2015_00000177
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:24 HICO_test2015_00000179
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:25 HICO_test2015_00000181
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:27 HICO_test2015_00000182
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:27 HICO_test2015_00000185
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:28 HICO_test2015_00000186
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:29 HICO_test2015_00000187
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:29 HICO_test2015_00000188
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:29 HICO_test2015_00000189
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:30 HICO_test2015_00000190
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:32 HICO_test2015_00000193
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:33 HICO_test2015_00000195
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:33 HICO_test2015_00000196
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:34 HICO_test2015_00000197
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:36 HICO_test2015_00000198
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:37 HICO_test2015_00000199
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:37 HICO_test2015_00000200
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:38 HICO_test2015_00000201
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:39 HICO_test2015_00000203
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:40 HICO_test2015_00000206
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:41 HICO_test2015_00000207
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:43 HICO_test2015_00000208
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:43 HICO_test2015_00000209
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:44 HICO_test2015_00000210
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:45 HICO_test2015_00000211
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:46 HICO_test2015_00000213
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:48 HICO_test2015_00000214
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:49 HICO_test2015_00000215
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:50 HICO_test2015_00000216
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:50 HICO_test2015_00000217
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:51 HICO_test2015_00000218
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:52 HICO_test2015_00000219
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:52 HICO_test2015_00000221
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:53 HICO_test2015_00000222
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:54 HICO_test2015_00000224
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:56 HICO_test2015_00000225
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:57 HICO_test2015_00000227
drwxrwxr-x   2 mona mona 4.0K Jan 12 00:58 HICO_test2015_00000231
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:00 HICO_test2015_00000232
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:00 HICO_test2015_00000233
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:00 HICO_test2015_00000234
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:01 HICO_test2015_00000236
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:01 HICO_test2015_00000237
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:02 HICO_test2015_00000238
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:03 HICO_test2015_00000239
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:04 HICO_test2015_00000240
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:05 HICO_test2015_00000243
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:05 HICO_test2015_00000245
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:06 HICO_test2015_00000246
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:07 HICO_test2015_00000248
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:07 HICO_test2015_00000251
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:08 HICO_test2015_00000252
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:09 HICO_test2015_00000253
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:10 HICO_test2015_00000254
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:12 HICO_test2015_00000257
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:14 HICO_test2015_00000258
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:15 HICO_test2015_00000259
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:16 HICO_test2015_00000260
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:16 HICO_test2015_00000261
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:17 HICO_test2015_00000262
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:19 HICO_test2015_00000264
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:20 HICO_test2015_00000266
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:21 HICO_test2015_00000267
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:22 HICO_test2015_00000268
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:22 HICO_test2015_00000269
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:23 HICO_test2015_00000270
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:25 HICO_test2015_00000272
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:27 HICO_test2015_00000274
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:29 HICO_test2015_00000275
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:30 HICO_test2015_00000276
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:32 HICO_test2015_00000277
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:34 HICO_test2015_00000278
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:35 HICO_test2015_00000279
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:37 HICO_test2015_00000281
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:38 HICO_test2015_00000282
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:40 HICO_test2015_00000283
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:41 HICO_test2015_00000284
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:42 HICO_test2015_00000285
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:43 HICO_test2015_00000286
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:43 HICO_test2015_00000288
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:45 HICO_test2015_00000289
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:45 HICO_test2015_00000290
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:46 HICO_test2015_00000291
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:47 HICO_test2015_00000293
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:48 HICO_test2015_00000294
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:50 HICO_test2015_00000295
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:51 HICO_test2015_00000296
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:54 HICO_test2015_00000299
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:56 HICO_test2015_00000300
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:58 HICO_test2015_00000301
drwxrwxr-x   2 mona mona 4.0K Jan 12 01:59 HICO_test2015_00000302
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:01 HICO_test2015_00000303
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:01 HICO_test2015_00000304
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:03 HICO_test2015_00000305
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:03 HICO_test2015_00000306
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:04 HICO_test2015_00000307
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:05 HICO_test2015_00000309
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:06 HICO_test2015_00000311
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:09 HICO_test2015_00000314
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:09 HICO_test2015_00000316
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:10 HICO_test2015_00000317
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:11 HICO_test2015_00000320
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:12 HICO_test2015_00000326
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:13 HICO_test2015_00000327
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:14 HICO_test2015_00000329
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:15 HICO_test2015_00000330
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:16 HICO_test2015_00000332
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:18 HICO_test2015_00000333
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:19 HICO_test2015_00000334
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:20 HICO_test2015_00000335
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:21 HICO_test2015_00000336
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:23 HICO_test2015_00000337
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:25 HICO_test2015_00000338
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:27 HICO_test2015_00000339
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:28 HICO_test2015_00000340
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:29 HICO_test2015_00000341
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:29 HICO_test2015_00000342
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:30 HICO_test2015_00000343
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:31 HICO_test2015_00000345
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:32 HICO_test2015_00000346
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:32 HICO_test2015_00000347
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:35 HICO_test2015_00000348
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:36 HICO_test2015_00000350
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:38 HICO_test2015_00000351
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:38 HICO_test2015_00000352
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:39 HICO_test2015_00000353
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:40 HICO_test2015_00000356
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:40 HICO_test2015_00000357
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:41 HICO_test2015_00000358
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:42 HICO_test2015_00000359
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:43 HICO_test2015_00000360
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:43 HICO_test2015_00000361
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:44 HICO_test2015_00000363
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:44 HICO_test2015_00000364
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:46 HICO_test2015_00000365
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:47 HICO_test2015_00000367
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:48 HICO_test2015_00000368
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:50 HICO_test2015_00000370
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:51 HICO_test2015_00000371
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:51 HICO_test2015_00000372
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:51 HICO_test2015_00000373
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:53 HICO_test2015_00000374
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:55 HICO_test2015_00000377
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:57 HICO_test2015_00000378
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:58 HICO_test2015_00000380
drwxrwxr-x   2 mona mona 4.0K Jan 12 02:59 HICO_test2015_00000381
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:00 HICO_test2015_00000383
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:02 HICO_test2015_00000384
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:03 HICO_test2015_00000386
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:05 HICO_test2015_00000387
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:06 HICO_test2015_00000388
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:07 HICO_test2015_00000389
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:09 HICO_test2015_00000390
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:09 HICO_test2015_00000391
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:10 HICO_test2015_00000395
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:10 HICO_test2015_00000396
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:12 HICO_test2015_00000397
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:12 HICO_test2015_00000398
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:14 HICO_test2015_00000399
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:14 HICO_test2015_00000400
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:15 HICO_test2015_00000401
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:16 HICO_test2015_00000402
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:17 HICO_test2015_00000404
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:17 HICO_test2015_00000406
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:19 HICO_test2015_00000408
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:20 HICO_test2015_00000409
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:22 HICO_test2015_00000411
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:23 HICO_test2015_00000413
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:25 HICO_test2015_00000414
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:28 HICO_test2015_00000415
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:28 HICO_test2015_00000416
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:29 HICO_test2015_00000417
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:30 HICO_test2015_00000420
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:31 HICO_test2015_00000422
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:32 HICO_test2015_00000424
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:33 HICO_test2015_00000425
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:34 HICO_test2015_00000427
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:35 HICO_test2015_00000428
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:35 HICO_test2015_00000429
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:35 HICO_test2015_00000430
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:36 HICO_test2015_00000431
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:36 HICO_test2015_00000432
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:37 HICO_test2015_00000434
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:38 HICO_test2015_00000435
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:41 HICO_test2015_00000437
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:42 HICO_test2015_00000438
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:43 HICO_test2015_00000439
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:44 HICO_test2015_00000440
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:45 HICO_test2015_00000441
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:45 HICO_test2015_00000442
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:46 HICO_test2015_00000443
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:47 HICO_test2015_00000444
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:48 HICO_test2015_00000445
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:48 HICO_test2015_00000447
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:49 HICO_test2015_00000448
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:49 HICO_test2015_00000449
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:50 HICO_test2015_00000450
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:50 HICO_test2015_00000451
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:51 HICO_test2015_00000452
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:52 HICO_test2015_00000453
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:53 HICO_test2015_00000454
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:54 HICO_test2015_00000455
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:55 HICO_test2015_00000456
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:56 HICO_test2015_00000457
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:57 HICO_test2015_00000459
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:57 HICO_test2015_00000460
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:59 HICO_test2015_00000464
drwxrwxr-x   2 mona mona 4.0K Jan 12 03:59 HICO_test2015_00000466
drwxrwxr-x   2 mona mona 4.0K Jan 12 04:00 HICO_test2015_00000468
drwxrwxr-x   2 mona mona 4.0K Jan 12 04:00 HICO_test2015_00000469
drwxrwxr-x   2 mona mona 4.0K Jan 12 04:01 HICO_test2015_00000470
drwxrwxr-x   2 mona mona 4.0K Jan 12 04:01 HICO_test2015_00000471
drwxrwxr-x 340 mona mona  20K Jan 12 04:01 .

And this is the set of images processed on server before both crashed (both processed 293 images):

~/mona/data/smplify-x/djrn_train_results/meshes$ ls
total 0
drwxrwxr-x 1 mona mona  56 Jan 11 16:45 ..
drwxrwxr-x 1 mona mona  14 Jan 11 16:46 HICO_train2015_00000001
drwxrwxr-x 1 mona mona  42 Jan 11 16:49 HICO_train2015_00000004
drwxrwxr-x 1 mona mona  14 Jan 11 16:49 HICO_train2015_00000005
drwxrwxr-x 1 mona mona  14 Jan 11 16:50 HICO_train2015_00000006
drwxrwxr-x 1 mona mona  28 Jan 11 16:52 HICO_train2015_00000009
drwxrwxr-x 1 mona mona  42 Jan 11 16:55 HICO_train2015_00000010
drwxrwxr-x 1 mona mona  14 Jan 11 16:56 HICO_train2015_00000011
drwxrwxr-x 1 mona mona  28 Jan 11 16:58 HICO_train2015_00000012
drwxrwxr-x 1 mona mona  14 Jan 11 16:58 HICO_train2015_00000015
drwxrwxr-x 1 mona mona  28 Jan 11 17:00 HICO_train2015_00000016
drwxrwxr-x 1 mona mona  42 Jan 11 17:02 HICO_train2015_00000017
drwxrwxr-x 1 mona mona  14 Jan 11 17:02 HICO_train2015_00000018
drwxrwxr-x 1 mona mona  14 Jan 11 17:03 HICO_train2015_00000019
drwxrwxr-x 1 mona mona  14 Jan 11 17:04 HICO_train2015_00000020
drwxrwxr-x 1 mona mona  14 Jan 11 17:05 HICO_train2015_00000022
drwxrwxr-x 1 mona mona  14 Jan 11 17:05 HICO_train2015_00000023
drwxrwxr-x 1 mona mona  28 Jan 11 17:07 HICO_train2015_00000024
drwxrwxr-x 1 mona mona  14 Jan 11 17:07 HICO_train2015_00000027
drwxrwxr-x 1 mona mona  14 Jan 11 17:08 HICO_train2015_00000028
drwxrwxr-x 1 mona mona  28 Jan 11 17:10 HICO_train2015_00000029
drwxrwxr-x 1 mona mona  28 Jan 11 17:12 HICO_train2015_00000030
drwxrwxr-x 1 mona mona  42 Jan 11 17:15 HICO_train2015_00000034
drwxrwxr-x 1 mona mona  28 Jan 11 17:16 HICO_train2015_00000035
drwxrwxr-x 1 mona mona  42 Jan 11 17:18 HICO_train2015_00000036
drwxrwxr-x 1 mona mona  42 Jan 11 17:20 HICO_train2015_00000038
drwxrwxr-x 1 mona mona  28 Jan 11 17:23 HICO_train2015_00000040
drwxrwxr-x 1 mona mona  42 Jan 11 17:25 HICO_train2015_00000041
drwxrwxr-x 1 mona mona  14 Jan 11 17:25 HICO_train2015_00000043
drwxrwxr-x 1 mona mona  42 Jan 11 17:28 HICO_train2015_00000044
drwxrwxr-x 1 mona mona  14 Jan 11 17:29 HICO_train2015_00000045
drwxrwxr-x 1 mona mona  42 Jan 11 17:31 HICO_train2015_00000046
drwxrwxr-x 1 mona mona  42 Jan 11 17:33 HICO_train2015_00000047
drwxrwxr-x 1 mona mona  14 Jan 11 17:34 HICO_train2015_00000050
drwxrwxr-x 1 mona mona  42 Jan 11 17:35 HICO_train2015_00000051
drwxrwxr-x 1 mona mona  42 Jan 11 17:37 HICO_train2015_00000053
drwxrwxr-x 1 mona mona  14 Jan 11 17:38 HICO_train2015_00000054
drwxrwxr-x 1 mona mona  14 Jan 11 17:39 HICO_train2015_00000055
drwxrwxr-x 1 mona mona  14 Jan 11 17:39 HICO_train2015_00000057
drwxrwxr-x 1 mona mona  14 Jan 11 17:40 HICO_train2015_00000058
drwxrwxr-x 1 mona mona  28 Jan 11 17:42 HICO_train2015_00000059
drwxrwxr-x 1 mona mona  42 Jan 11 17:44 HICO_train2015_00000060
drwxrwxr-x 1 mona mona  42 Jan 11 17:47 HICO_train2015_00000061
drwxrwxr-x 1 mona mona  42 Jan 11 17:50 HICO_train2015_00000062
drwxrwxr-x 1 mona mona  42 Jan 11 17:52 HICO_train2015_00000063
drwxrwxr-x 1 mona mona  42 Jan 11 17:54 HICO_train2015_00000065
drwxrwxr-x 1 mona mona  14 Jan 11 17:55 HICO_train2015_00000067
drwxrwxr-x 1 mona mona  14 Jan 11 17:57 HICO_train2015_00000068
drwxrwxr-x 1 mona mona  42 Jan 11 17:59 HICO_train2015_00000069
drwxrwxr-x 1 mona mona  14 Jan 11 18:00 HICO_train2015_00000070
drwxrwxr-x 1 mona mona  28 Jan 11 18:01 HICO_train2015_00000071
drwxrwxr-x 1 mona mona  14 Jan 11 18:02 HICO_train2015_00000072
drwxrwxr-x 1 mona mona  28 Jan 11 18:03 HICO_train2015_00000073
drwxrwxr-x 1 mona mona  14 Jan 11 18:03 HICO_train2015_00000074
drwxrwxr-x 1 mona mona  42 Jan 11 18:05 HICO_train2015_00000075
drwxrwxr-x 1 mona mona  42 Jan 11 18:08 HICO_train2015_00000076
drwxrwxr-x 1 mona mona  42 Jan 11 18:10 HICO_train2015_00000077
drwxrwxr-x 1 mona mona  14 Jan 11 18:10 HICO_train2015_00000078
drwxrwxr-x 1 mona mona  14 Jan 11 18:11 HICO_train2015_00000079
drwxrwxr-x 1 mona mona  42 Jan 11 18:12 HICO_train2015_00000081
drwxrwxr-x 1 mona mona  14 Jan 11 18:13 HICO_train2015_00000082
drwxrwxr-x 1 mona mona  42 Jan 11 18:15 HICO_train2015_00000086
drwxrwxr-x 1 mona mona  28 Jan 11 18:16 HICO_train2015_00000088
drwxrwxr-x 1 mona mona  14 Jan 11 18:17 HICO_train2015_00000089
drwxrwxr-x 1 mona mona  42 Jan 11 18:18 HICO_train2015_00000093
drwxrwxr-x 1 mona mona  14 Jan 11 18:19 HICO_train2015_00000094
drwxrwxr-x 1 mona mona  28 Jan 11 18:21 HICO_train2015_00000096
drwxrwxr-x 1 mona mona  14 Jan 11 18:21 HICO_train2015_00000098
drwxrwxr-x 1 mona mona  14 Jan 11 18:22 HICO_train2015_00000099
drwxrwxr-x 1 mona mona  28 Jan 11 18:24 HICO_train2015_00000100
drwxrwxr-x 1 mona mona  42 Jan 11 18:27 HICO_train2015_00000102
drwxrwxr-x 1 mona mona  28 Jan 11 18:29 HICO_train2015_00000103
drwxrwxr-x 1 mona mona  14 Jan 11 18:29 HICO_train2015_00000104
drwxrwxr-x 1 mona mona  14 Jan 11 18:30 HICO_train2015_00000105
drwxrwxr-x 1 mona mona  42 Jan 11 18:33 HICO_train2015_00000106
drwxrwxr-x 1 mona mona  14 Jan 11 18:34 HICO_train2015_00000107
drwxrwxr-x 1 mona mona  14 Jan 11 18:35 HICO_train2015_00000108
drwxrwxr-x 1 mona mona  14 Jan 11 18:36 HICO_train2015_00000112
drwxrwxr-x 1 mona mona  14 Jan 11 18:37 HICO_train2015_00000114
drwxrwxr-x 1 mona mona  28 Jan 11 18:38 HICO_train2015_00000115
drwxrwxr-x 1 mona mona  42 Jan 11 18:40 HICO_train2015_00000116
drwxrwxr-x 1 mona mona  14 Jan 11 18:41 HICO_train2015_00000118
drwxrwxr-x 1 mona mona  14 Jan 11 18:41 HICO_train2015_00000119
drwxrwxr-x 1 mona mona  28 Jan 11 18:43 HICO_train2015_00000120
drwxrwxr-x 1 mona mona  14 Jan 11 18:44 HICO_train2015_00000122
drwxrwxr-x 1 mona mona  42 Jan 11 18:45 HICO_train2015_00000125
drwxrwxr-x 1 mona mona  28 Jan 11 18:47 HICO_train2015_00000126
drwxrwxr-x 1 mona mona  14 Jan 11 18:48 HICO_train2015_00000127
drwxrwxr-x 1 mona mona  42 Jan 11 18:49 HICO_train2015_00000128
drwxrwxr-x 1 mona mona  28 Jan 11 18:51 HICO_train2015_00000130
drwxrwxr-x 1 mona mona  14 Jan 11 18:52 HICO_train2015_00000131
drwxrwxr-x 1 mona mona  28 Jan 11 18:53 HICO_train2015_00000132
drwxrwxr-x 1 mona mona  42 Jan 11 18:57 HICO_train2015_00000133
drwxrwxr-x 1 mona mona  28 Jan 11 18:59 HICO_train2015_00000134
drwxrwxr-x 1 mona mona  42 Jan 11 19:02 HICO_train2015_00000135
drwxrwxr-x 1 mona mona  28 Jan 11 19:03 HICO_train2015_00000136
drwxrwxr-x 1 mona mona  14 Jan 11 19:04 HICO_train2015_00000137
drwxrwxr-x 1 mona mona  14 Jan 11 19:05 HICO_train2015_00000138
drwxrwxr-x 1 mona mona  14 Jan 11 19:05 HICO_train2015_00000139
drwxrwxr-x 1 mona mona  14 Jan 11 19:07 HICO_train2015_00000140
drwxrwxr-x 1 mona mona  14 Jan 11 19:08 HICO_train2015_00000144
drwxrwxr-x 1 mona mona  14 Jan 11 19:09 HICO_train2015_00000145
drwxrwxr-x 1 mona mona  14 Jan 11 19:10 HICO_train2015_00000149
drwxrwxr-x 1 mona mona  14 Jan 11 19:11 HICO_train2015_00000152
drwxrwxr-x 1 mona mona  14 Jan 11 19:11 HICO_train2015_00000154
drwxrwxr-x 1 mona mona  14 Jan 11 19:12 HICO_train2015_00000155
drwxrwxr-x 1 mona mona  14 Jan 11 19:13 HICO_train2015_00000157
drwxrwxr-x 1 mona mona  14 Jan 11 19:14 HICO_train2015_00000158
drwxrwxr-x 1 mona mona  14 Jan 11 19:15 HICO_train2015_00000165
drwxrwxr-x 1 mona mona  28 Jan 11 19:16 HICO_train2015_00000166
drwxrwxr-x 1 mona mona  42 Jan 11 19:19 HICO_train2015_00000168
drwxrwxr-x 1 mona mona  42 Jan 11 19:21 HICO_train2015_00000170
drwxrwxr-x 1 mona mona  28 Jan 11 19:22 HICO_train2015_00000173
drwxrwxr-x 1 mona mona  14 Jan 11 19:23 HICO_train2015_00000174
drwxrwxr-x 1 mona mona  42 Jan 11 19:25 HICO_train2015_00000175
drwxrwxr-x 1 mona mona  14 Jan 11 19:26 HICO_train2015_00000177
drwxrwxr-x 1 mona mona  42 Jan 11 19:28 HICO_train2015_00000178
drwxrwxr-x 1 mona mona  42 Jan 11 19:30 HICO_train2015_00000179
drwxrwxr-x 1 mona mona  14 Jan 11 19:30 HICO_train2015_00000181
drwxrwxr-x 1 mona mona  28 Jan 11 19:31 HICO_train2015_00000183
drwxrwxr-x 1 mona mona  28 Jan 11 19:32 HICO_train2015_00000184
drwxrwxr-x 1 mona mona  14 Jan 11 19:33 HICO_train2015_00000185
drwxrwxr-x 1 mona mona  14 Jan 11 19:34 HICO_train2015_00000188
drwxrwxr-x 1 mona mona  14 Jan 11 19:35 HICO_train2015_00000189
drwxrwxr-x 1 mona mona  42 Jan 11 19:36 HICO_train2015_00000190
drwxrwxr-x 1 mona mona  14 Jan 11 19:37 HICO_train2015_00000192
drwxrwxr-x 1 mona mona  42 Jan 11 19:41 HICO_train2015_00000194
drwxrwxr-x 1 mona mona  14 Jan 11 19:42 HICO_train2015_00000195
drwxrwxr-x 1 mona mona  42 Jan 11 19:45 HICO_train2015_00000196
drwxrwxr-x 1 mona mona  14 Jan 11 19:45 HICO_train2015_00000198
drwxrwxr-x 1 mona mona  28 Jan 11 19:47 HICO_train2015_00000199
drwxrwxr-x 1 mona mona  42 Jan 11 19:49 HICO_train2015_00000200
drwxrwxr-x 1 mona mona  14 Jan 11 19:49 HICO_train2015_00000201
drwxrwxr-x 1 mona mona  14 Jan 11 19:50 HICO_train2015_00000202
drwxrwxr-x 1 mona mona  14 Jan 11 19:51 HICO_train2015_00000203
drwxrwxr-x 1 mona mona  14 Jan 11 19:52 HICO_train2015_00000205
drwxrwxr-x 1 mona mona  42 Jan 11 19:54 HICO_train2015_00000206
drwxrwxr-x 1 mona mona  14 Jan 11 19:55 HICO_train2015_00000208
drwxrwxr-x 1 mona mona  28 Jan 11 19:56 HICO_train2015_00000210
drwxrwxr-x 1 mona mona  14 Jan 11 19:57 HICO_train2015_00000211
drwxrwxr-x 1 mona mona  14 Jan 11 19:58 HICO_train2015_00000212
drwxrwxr-x 1 mona mona  42 Jan 11 20:00 HICO_train2015_00000213
drwxrwxr-x 1 mona mona  28 Jan 11 20:02 HICO_train2015_00000215
drwxrwxr-x 1 mona mona  14 Jan 11 20:03 HICO_train2015_00000217
drwxrwxr-x 1 mona mona  14 Jan 11 20:04 HICO_train2015_00000218
drwxrwxr-x 1 mona mona  42 Jan 11 20:06 HICO_train2015_00000219
drwxrwxr-x 1 mona mona  14 Jan 11 20:07 HICO_train2015_00000220
drwxrwxr-x 1 mona mona  14 Jan 11 20:07 HICO_train2015_00000222
drwxrwxr-x 1 mona mona  14 Jan 11 20:08 HICO_train2015_00000224
drwxrwxr-x 1 mona mona  14 Jan 11 20:09 HICO_train2015_00000225
drwxrwxr-x 1 mona mona  28 Jan 11 20:10 HICO_train2015_00000228
drwxrwxr-x 1 mona mona  14 Jan 11 20:11 HICO_train2015_00000229
drwxrwxr-x 1 mona mona  14 Jan 11 20:13 HICO_train2015_00000231
drwxrwxr-x 1 mona mona  28 Jan 11 20:14 HICO_train2015_00000232
drwxrwxr-x 1 mona mona  14 Jan 11 20:15 HICO_train2015_00000234
drwxrwxr-x 1 mona mona  28 Jan 11 20:16 HICO_train2015_00000236
drwxrwxr-x 1 mona mona  14 Jan 11 20:17 HICO_train2015_00000237
drwxrwxr-x 1 mona mona  14 Jan 11 20:18 HICO_train2015_00000238
drwxrwxr-x 1 mona mona  28 Jan 11 20:19 HICO_train2015_00000239
drwxrwxr-x 1 mona mona  42 Jan 11 20:21 HICO_train2015_00000242
drwxrwxr-x 1 mona mona  14 Jan 11 20:21 HICO_train2015_00000243
drwxrwxr-x 1 mona mona  14 Jan 11 20:21 HICO_train2015_00000244
drwxrwxr-x 1 mona mona  42 Jan 11 20:23 HICO_train2015_00000245
drwxrwxr-x 1 mona mona  42 Jan 11 20:27 HICO_train2015_00000246
drwxrwxr-x 1 mona mona  14 Jan 11 20:28 HICO_train2015_00000247
drwxrwxr-x 1 mona mona  14 Jan 11 20:28 HICO_train2015_00000249
drwxrwxr-x 1 mona mona  42 Jan 11 20:30 HICO_train2015_00000250
drwxrwxr-x 1 mona mona  14 Jan 11 20:32 HICO_train2015_00000251
drwxrwxr-x 1 mona mona  14 Jan 11 20:33 HICO_train2015_00000253
drwxrwxr-x 1 mona mona  28 Jan 11 20:34 HICO_train2015_00000255
drwxrwxr-x 1 mona mona  14 Jan 11 20:36 HICO_train2015_00000256
drwxrwxr-x 1 mona mona  14 Jan 11 20:37 HICO_train2015_00000258
drwxrwxr-x 1 mona mona  28 Jan 11 20:38 HICO_train2015_00000259
drwxrwxr-x 1 mona mona  28 Jan 11 20:39 HICO_train2015_00000261
drwxrwxr-x 1 mona mona  14 Jan 11 20:40 HICO_train2015_00000262
drwxrwxr-x 1 mona mona  14 Jan 11 20:41 HICO_train2015_00000264
drwxrwxr-x 1 mona mona  14 Jan 11 20:41 HICO_train2015_00000266
drwxrwxr-x 1 mona mona  42 Jan 11 20:43 HICO_train2015_00000269
drwxrwxr-x 1 mona mona  42 Jan 11 20:46 HICO_train2015_00000270
drwxrwxr-x 1 mona mona  14 Jan 11 20:47 HICO_train2015_00000271
drwxrwxr-x 1 mona mona  14 Jan 11 20:47 HICO_train2015_00000272
drwxrwxr-x 1 mona mona  42 Jan 11 20:49 HICO_train2015_00000273
drwxrwxr-x 1 mona mona  42 Jan 11 20:51 HICO_train2015_00000274
drwxrwxr-x 1 mona mona  42 Jan 11 20:54 HICO_train2015_00000276
drwxrwxr-x 1 mona mona  42 Jan 11 20:56 HICO_train2015_00000278
drwxrwxr-x 1 mona mona  14 Jan 11 20:58 HICO_train2015_00000279
drwxrwxr-x 1 mona mona  28 Jan 11 21:00 HICO_train2015_00000280
drwxrwxr-x 1 mona mona  14 Jan 11 21:01 HICO_train2015_00000281
drwxrwxr-x 1 mona mona  28 Jan 11 21:02 HICO_train2015_00000282
drwxrwxr-x 1 mona mona  42 Jan 11 21:04 HICO_train2015_00000283
drwxrwxr-x 1 mona mona  28 Jan 11 21:06 HICO_train2015_00000284
drwxrwxr-x 1 mona mona  42 Jan 11 21:08 HICO_train2015_00000285
drwxrwxr-x 1 mona mona  14 Jan 11 21:09 HICO_train2015_00000286
drwxrwxr-x 1 mona mona  28 Jan 11 21:10 HICO_train2015_00000287
drwxrwxr-x 1 mona mona  14 Jan 11 21:10 HICO_train2015_00000289
drwxrwxr-x 1 mona mona  28 Jan 11 21:12 HICO_train2015_00000290
drwxrwxr-x 1 mona mona  28 Jan 11 21:14 HICO_train2015_00000291
drwxrwxr-x 1 mona mona  42 Jan 11 21:16 HICO_train2015_00000292
drwxrwxr-x 1 mona mona  42 Jan 11 21:18 HICO_train2015_00000296
drwxrwxr-x 1 mona mona  28 Jan 11 21:19 HICO_train2015_00000297
drwxrwxr-x 1 mona mona  28 Jan 11 21:20 HICO_train2015_00000298
drwxrwxr-x 1 mona mona  28 Jan 11 21:24 HICO_train2015_00000299
drwxrwxr-x 1 mona mona  14 Jan 11 21:24 HICO_train2015_00000300
drwxrwxr-x 1 mona mona  14 Jan 11 21:25 HICO_train2015_00000301
drwxrwxr-x 1 mona mona  14 Jan 11 21:25 HICO_train2015_00000302
drwxrwxr-x 1 mona mona  42 Jan 11 21:27 HICO_train2015_00000303
drwxrwxr-x 1 mona mona  14 Jan 11 21:28 HICO_train2015_00000305
drwxrwxr-x 1 mona mona  14 Jan 11 21:28 HICO_train2015_00000307
drwxrwxr-x 1 mona mona  14 Jan 11 21:29 HICO_train2015_00000308
drwxrwxr-x 1 mona mona  14 Jan 11 21:30 HICO_train2015_00000309
drwxrwxr-x 1 mona mona  14 Jan 11 21:30 HICO_train2015_00000312
drwxrwxr-x 1 mona mona  28 Jan 11 21:31 HICO_train2015_00000313
drwxrwxr-x 1 mona mona  28 Jan 11 21:33 HICO_train2015_00000314
drwxrwxr-x 1 mona mona  42 Jan 11 21:35 HICO_train2015_00000317
drwxrwxr-x 1 mona mona  42 Jan 11 21:38 HICO_train2015_00000318
drwxrwxr-x 1 mona mona  28 Jan 11 21:40 HICO_train2015_00000319
drwxrwxr-x 1 mona mona  28 Jan 11 21:41 HICO_train2015_00000320
drwxrwxr-x 1 mona mona  14 Jan 11 21:42 HICO_train2015_00000322
drwxrwxr-x 1 mona mona  14 Jan 11 21:43 HICO_train2015_00000323
drwxrwxr-x 1 mona mona  28 Jan 11 21:45 HICO_train2015_00000324
drwxrwxr-x 1 mona mona  14 Jan 11 21:45 HICO_train2015_00000325
drwxrwxr-x 1 mona mona  14 Jan 11 21:46 HICO_train2015_00000326
drwxrwxr-x 1 mona mona  28 Jan 11 21:47 HICO_train2015_00000327
drwxrwxr-x 1 mona mona  14 Jan 11 21:48 HICO_train2015_00000329
drwxrwxr-x 1 mona mona  14 Jan 11 21:48 HICO_train2015_00000333
drwxrwxr-x 1 mona mona  42 Jan 11 21:49 HICO_train2015_00000335
drwxrwxr-x 1 mona mona  14 Jan 11 21:50 HICO_train2015_00000336
drwxrwxr-x 1 mona mona  14 Jan 11 21:52 HICO_train2015_00000337
drwxrwxr-x 1 mona mona  14 Jan 11 21:53 HICO_train2015_00000338
drwxrwxr-x 1 mona mona  14 Jan 11 21:53 HICO_train2015_00000339
drwxrwxr-x 1 mona mona  28 Jan 11 21:54 HICO_train2015_00000340
drwxrwxr-x 1 mona mona  14 Jan 11 21:55 HICO_train2015_00000341
drwxrwxr-x 1 mona mona  42 Jan 11 21:57 HICO_train2015_00000342
drwxrwxr-x 1 mona mona  42 Jan 11 22:00 HICO_train2015_00000344
drwxrwxr-x 1 mona mona  42 Jan 11 22:03 HICO_train2015_00000345
drwxrwxr-x 1 mona mona  14 Jan 11 22:04 HICO_train2015_00000346
drwxrwxr-x 1 mona mona  14 Jan 11 22:05 HICO_train2015_00000347
drwxrwxr-x 1 mona mona  42 Jan 11 22:07 HICO_train2015_00000348
drwxrwxr-x 1 mona mona  14 Jan 11 22:07 HICO_train2015_00000349
drwxrwxr-x 1 mona mona  42 Jan 11 22:09 HICO_train2015_00000350
drwxrwxr-x 1 mona mona  14 Jan 11 22:10 HICO_train2015_00000352
drwxrwxr-x 1 mona mona  14 Jan 11 22:11 HICO_train2015_00000353
drwxrwxr-x 1 mona mona  28 Jan 11 22:12 HICO_train2015_00000354
drwxrwxr-x 1 mona mona  14 Jan 11 22:13 HICO_train2015_00000355
drwxrwxr-x 1 mona mona  14 Jan 11 22:13 HICO_train2015_00000356
drwxrwxr-x 1 mona mona  28 Jan 11 22:15 HICO_train2015_00000357
drwxrwxr-x 1 mona mona  28 Jan 11 22:16 HICO_train2015_00000358
drwxrwxr-x 1 mona mona  28 Jan 11 22:17 HICO_train2015_00000360
drwxrwxr-x 1 mona mona  42 Jan 11 22:19 HICO_train2015_00000361
drwxrwxr-x 1 mona mona  14 Jan 11 22:20 HICO_train2015_00000362
drwxrwxr-x 1 mona mona  14 Jan 11 22:21 HICO_train2015_00000363
drwxrwxr-x 1 mona mona  14 Jan 11 22:22 HICO_train2015_00000364
drwxrwxr-x 1 mona mona  14 Jan 11 22:22 HICO_train2015_00000365
drwxrwxr-x 1 mona mona  14 Jan 11 22:23 HICO_train2015_00000366
drwxrwxr-x 1 mona mona  42 Jan 11 22:25 HICO_train2015_00000368
drwxrwxr-x 1 mona mona  14 Jan 11 22:26 HICO_train2015_00000369
drwxrwxr-x 1 mona mona  14 Jan 11 22:26 HICO_train2015_00000372
drwxrwxr-x 1 mona mona  42 Jan 11 22:28 HICO_train2015_00000374
drwxrwxr-x 1 mona mona  14 Jan 11 22:29 HICO_train2015_00000376
drwxrwxr-x 1 mona mona  42 Jan 11 22:31 HICO_train2015_00000378
drwxrwxr-x 1 mona mona  14 Jan 11 22:32 HICO_train2015_00000379
drwxrwxr-x 1 mona mona  42 Jan 11 22:34 HICO_train2015_00000380
drwxrwxr-x 1 mona mona  14 Jan 11 22:35 HICO_train2015_00000381
drwxrwxr-x 1 mona mona  14 Jan 11 22:36 HICO_train2015_00000382
drwxrwxr-x 1 mona mona  14 Jan 11 22:37 HICO_train2015_00000384
drwxrwxr-x 1 mona mona  14 Jan 11 22:37 HICO_train2015_00000385
drwxrwxr-x 1 mona mona  42 Jan 11 22:39 HICO_train2015_00000386
drwxrwxr-x 1 mona mona  14 Jan 11 22:40 HICO_train2015_00000388
drwxrwxr-x 1 mona mona  14 Jan 11 22:41 HICO_train2015_00000389
drwxrwxr-x 1 mona mona  42 Jan 11 22:44 HICO_train2015_00000390
drwxrwxr-x 1 mona mona  42 Jan 11 22:47 HICO_train2015_00000391
drwxrwxr-x 1 mona mona  14 Jan 11 22:48 HICO_train2015_00000392
drwxrwxr-x 1 mona mona  14 Jan 11 22:48 HICO_train2015_00000393
drwxrwxr-x 1 mona mona  14 Jan 11 22:49 HICO_train2015_00000394
drwxrwxr-x 1 mona mona  14 Jan 11 22:50 HICO_train2015_00000395
drwxrwxr-x 1 mona mona  14 Jan 11 22:51 HICO_train2015_00000396
drwxrwxr-x 1 mona mona  42 Jan 11 22:53 HICO_train2015_00000397
drwxrwxr-x 1 mona mona  28 Jan 11 22:54 HICO_train2015_00000401
drwxrwxr-x 1 mona mona  42 Jan 11 22:57 HICO_train2015_00000402
drwxrwxr-x 1 mona mona  14 Jan 11 22:58 HICO_train2015_00000403
drwxrwxr-x 1 mona mona  14 Jan 11 22:58 HICO_train2015_00000404
drwxrwxr-x 1 mona mona  14 Jan 11 22:59 HICO_train2015_00000406
drwxrwxr-x 1 mona mona  14 Jan 11 23:00 HICO_train2015_00000407
drwxrwxr-x 1 mona mona  42 Jan 11 23:01 HICO_train2015_00000408
drwxrwxr-x 1 mona mona  14 Jan 11 23:02 HICO_train2015_00000409
drwxrwxr-x 1 mona mona  14 Jan 11 23:03 HICO_train2015_00000410
drwxrwxr-x 1 mona mona  42 Jan 11 23:05 HICO_train2015_00000411
drwxrwxr-x 1 mona mona  14 Jan 11 23:06 HICO_train2015_00000412
drwxrwxr-x 1 mona mona  14 Jan 11 23:08 HICO_train2015_00000415
drwxrwxr-x 1 mona mona  28 Jan 11 23:09 HICO_train2015_00000416
drwxrwxr-x 1 mona mona  42 Jan 11 23:11 HICO_train2015_00000417
drwxrwxr-x 1 mona mona  14 Jan 11 23:11 HICO_train2015_00000418
drwxrwxr-x 1 mona mona  14 Jan 11 23:12 HICO_train2015_00000419
drwxrwxr-x 1 mona mona   0 Jan 11 23:12 HICO_train2015_00000420
monacv commented 3 years ago

also now I got it for this test image from HICO dataset:

It was able to process 933 images in the test set:

mona@ubuntu:~/mona/data/smplify-x/djrn_test_results/meshes$ numdir
933
Processing: ../../data/smplify-x/djrn_test_data/images/HICO_test2015_00001356.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 1.6837
Camera initialization final loss 4819.7524
Stage 000 done after 2.4872 seconds
Stage 001 done after 2.1198 seconds
Stage 002 done after 3.0530 seconds
Stage 003 done after 14.8807 seconds
Orientation:   0%|                                                                                           | 0/1 [00:22<?, ?it/sNaN loss value, stopping!███████████████████████████████████████████████████████████▏                 | 4/5 [00:22<00:07,  7.47s/it]
Stage 004 done after 25.5808 seconds
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:48<00:00,  9.63s/it]
Body fitting Orientation 0 done after 48.1341 seconds
Body final loss val = 2266.42139
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:48<00:00, 48.14s/it]
Processing: ../../data/smplify-x/djrn_test_data/images/HICO_test2015_00001357.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
NaN loss value, stopping!
Camera initialization done after 2.5745
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 262, in main
    **args)
  File "~/mona/code/smplify-x/smplifyx/fit_single_frame.py", line 367, in fit_single_frame
    cam_init_loss_val))
TypeError: unsupported format string passed to NoneType.__format__
Foruck commented 3 years ago

This is related to the optimizer used by SMPLify-X. The error usually occurs when the reconstruction is failed, according to our experience. We simply omit the images that cause these errors.

monacv commented 3 years ago

So, I was very curious and decided to run the SMPLify-X only for one of the images that we got a NAN loss for and I was able to produce meshes using SMPLify-X.

So I am very confused why do I get a NAN error when it is in a folder along with other images

Screenshot from 2021-01-13 21-22-30 Screenshot from 2021-01-13 21-21-38 Screenshot from 2021-01-13 21-21-25 Screenshot from 2021-01-13 21-21-08 Screenshot from 2021-01-13 21-25-55

Also, you mentioned that you skipped the images that had caused you problems. Could you either please share the code that does this or share the final set of train and test images that doesn't have these problems so I could reproduce your results?

Thanks a lot

Foruck commented 3 years ago

This is because of the LBGFS optimizer used by smplify-x. When it encounters a failure with NaN loss, the following reconstructions will also result in NaN loss even though it could have be successfully reconstructed. We will update the list of images that we successfully reconstructed.

monacv commented 3 years ago

@Foruck @DirtyHarryLYL you have closed this issue but the list of images for which SMPLIFY-X doesn't throw NaN loss error has not been released. Could you please provide the list of the images that were used so that I could reproduce your results? Thanks a lot.