I wonder how you trained Hourglass model...

[Indigo6]

I downloaded your project, tried to train Hourglass, but got following error:

=> creating output/coco/hourglass/hg4_128x96_d256x3_adam_lr2 => creating log/coco/hourglass/hg4_128x96_d256x3_adam_lr2_2021-08-16-12-55 Namespace(cfg='experiments/coco/hourglass/hg4_128x96_d256x3_adam_lr2.5e-4.yaml', dataDir='', logDir='', modelDir='', opts=[], prevModelDir='') AUTO_RESUME: True CUDNN: BENCHMARK: True DETERMINISTIC: False ENABLED: True DATASET: COLOR_RGB: False DATASET: coco DATA_FORMAT: jpg FLIP: True HYBRID_JOINTS_TYPE: NUM_JOINTS_HALF_BODY: 8 PROB_HALF_BODY: 0.0 ROOT: data/coco ROT_FACTOR: 40 SCALE_FACTOR: 0.3 SELECT_DATA: False TEST_SET: val2017 TRAIN_SET: train2017 DATA_DIR: DEBUG: DEBUG: True SAVE_BATCH_IMAGES_GT: True SAVE_BATCH_IMAGES_PRED: True SAVE_HEATMAPS_GT: True SAVE_HEATMAPS_PRED: True GPUS: (0,) LOG_DIR: log LOSS: TOPK: 8 USE_DIFFERENT_JOINTS_WEIGHT: False USE_OHKM: False USE_TARGET_WEIGHT: True MODEL: EXTRA: NUM_BLOCKS: 1 NUM_FEATURES: 256 NUM_STACKS: 4 HEATMAP_SIZE: [24, 32] IMAGE_SIZE: [96, 128] INIT_WEIGHTS: False NAME: hourglass NUM_JOINTS: 17 PRETRAINED: models/pytorch/imagenet/resnet50-19c8e357.pth SIGMA: 1 TAG_PER_JOINT: True TARGET_TYPE: gaussian OUTPUT_DIR: output PIN_MEMORY: True PRINT_FREQ: 100 RANK: 0 TEST: BATCH_SIZE_PER_GPU: 32 BBOX_THRE: 1.0 BLUR_KERNEL: 11 COCO_BBOX_FILE: data/coco/person_detection_results/COCO_val2017_detections_AP_H_56_person.json FLIP_TEST: True IMAGE_THRE: 0.0 IN_VIS_THRE: 0.2 MODEL_FILE: NMS_THRE: 1.0 OKS_THRE: 0.9 POST_PROCESS: True SOFT_NMS: False USE_GT_BBOX: True TRAIN: BATCH_SIZE_PER_GPU: 8 BEGIN_EPOCH: 0 CHECKPOINT: END_EPOCH: 140 GAMMA1: 0.99 GAMMA2: 0.0 LR: 0.00025 LR_FACTOR: 0.1 LR_STEP: [90, 120] MOMENTUM: 0.9 NESTEROV: False OPTIMIZER: adam RESUME: False SHUFFLE: True WD: 0.0001 WORKERS: 24 The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 3 Error occurs, No graph saved Traceback (most recent call last): File "/home/fl/dark/tools/train.py", line 223, in main() File "/home/fl/dark/tools/train.py", line 111, in main writer_dict['writer'].add_graph(model, (dump_input, )) File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/tensorboardX/writer.py", line 945, in add_graph self._get_file_writer().add_graph(graph(model, input_to_model, verbose)) File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 292, in graph raise e File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 286, in graph trace = torch.jit.trace(model, args) File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/jit/_trace.py", line 742, in trace _module_class, File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/jit/_trace.py", line 940, in trace_module _force_outplace, File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 887, in _call_impl result = self._slow_forward(*input, kwargs) File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward result = self.forward(*input, *kwargs) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 182, in forward y = self.hgi File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 887, in _call_impl result = self._slow_forward(input, kwargs) File "/home/fl/miniconda3/envs/pose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward result = self.forward(*input, **kwargs) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 95, in forward return self._hour_glass_forward(self.depth, x) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 86, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 86, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 86, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/fl/dark/tools/../lib/models/hourglass.py", line 91, in _hour_glass_forward out = up1 + up2 RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 3 Process finished with exit code 1

[Indigo6]

@hbin-ac

[xizero00]

Hi, @Indigo6 , We forget to change the hourglass model file in the public repository. I'm sorry for that, we have updated the hourglass.py now. If you find any question, please feel free to contact us.

[Klawens]

I still got this problem when training hourglass 256&384, 128 is okay. The size of tensor a (9) must match the size of tensor b (3) at non-singleton dimension 3 Error occurs, No graph saved Traceback (most recent call last): File "tools/train.py", line 223, in <module> main() File "tools/train.py", line 111, in main writer_dict['writer'].add_graph(model, (dump_input, )) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/tensorboardX/writer.py", line 945, in add_graph self._get_file_writer().add_graph(graph(model, input_to_model, verbose)) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 292, in graph raise e File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 286, in graph trace = torch.jit.trace(model, args) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/jit/_trace.py", line 733, in trace return trace_module( File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/jit/_trace.py", line 934, in trace_module module._c._create_method_from_trace( File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/nn/modules/module.py", line 887, in _call_impl result = self._slow_forward(*input, **kwargs) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward result = self.forward(*input, **kwargs) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 188, in forward y = self.hg[i](x) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/nn/modules/module.py", line 887, in _call_impl result = self._slow_forward(*input, **kwargs) File "/home/lsc/anaconda3/envs/dark/lib/python3.8/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward result = self.forward(*input, **kwargs) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 101, in forward return self._hour_glass_forward(self.depth, x) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 87, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 87, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 87, in _hour_glass_forward low2 = self._hour_glass_forward(n-1, low1) File "/home/lsc/pose/DarkPose/tools/../lib/models/hourglass.py", line 96, in _hour_glass_forward out = up1 + up2 RuntimeError: The size of tensor a (9) must match the size of tensor b (3) at non-singleton dimension 3