Training RGB video (ntu60 xsub)

katahiyu commented 1 year ago

Sorry for the multiple questions. I am now retraining the RGB video. I run the following command. python main_rgb_fused.py recognition -c config/ntu60_xsub/train_rgb_fused.yaml [--work_dir <work folder>] However, the accuracy at epoch 5 is low as shown in the image below. I would like to know if you know the cause of this problem.

Here is my train_rgb_fused.yaml file.

# python main_rgb_fused.py recognition -c config/ntu60_xsub/train_rgb_fused.yaml
# work_dir: ../../data/st-gcn/xsub/with_rgb/rgb_tp15_tf5_
work_dir: work_dir/ntu60/xsub/rgb_fused

skeleton_joints_pkl: results/ntu60/xsub/joint_result_msg3d.pkl
skeleton_bones_pkl: results/ntu60/xsub/bone_result_msg3d.pkl
# skeleton_joints_pkl: results/ntu60/xsub/joint_result_stgcn.pkl
# skeleton_bones_pkl: results/ntu60/xsub/bone_result_stgcn.pkl
# skeleton_joints_pkl: work_dir/ntu60/xsub/skeleton_joint/test_result_epoch25.pkl
# skeleton_bones_pkl: work_dir/ntu60/xsub/skeleton_bone/test_result_epoch55.pkl

# feeder
feeder: feeder.feeder_rgb_fused_ntu.Feeder
train_feeder_args:
  debug: False
  random_choose: False
  centralization: False
  random_move: False
  window_size: -1
  random_flip: False
  random_interval: True
  temporal_rgb_frames: 5
  # data_path: /media/bruce/2Tssd/data/ntu/xsub/train_data_joint.npy
  # label_path: /media/bruce/2Tssd/data/ntu/xsub/train_label.pkl
  data_path: data/ntu/xsub/train_data_joint.npy
  label_path: data/ntu/xsub/train_label.pkl
test_feeder_args:
  debug: False
  centralization: False
  evaluation: True
  temporal_rgb_frames: 5
  # data_path: /media/bruce/2Tssd/data/ntu/xsub/val_data_joint.npy
  # label_path: /media/bruce/2Tssd/data/ntu/xsub/val_label.pkl
  data_path: data/ntu/xsub/val_data_joint.npy
  label_path: data/ntu/xsub/val_label.pkl

# model
model: net.mmn.Model
model_args:
  in_channels: 3
  num_class: 60
  dropout: 0.5
  edge_importance_weighting: True
  graph_args:
    layout: 'ntu-rgb+d'
    strategy: 'spatial'

# training
temporal_positions: 15
fix_weights: True
joint_weights: models/ntu60/xsub/joint_model_stgcn.pt
# joint_weights: work_dir/ntu60/xsub/skeleton_joint/epoch25_model.pt
device: [0,1,2,3]
weight_decay: 0.0001
base_lr: 0.1
step: [10, 50]
batch_size: 32
test_batch_size: 32
num_epoch: 80

# debug
debug: False

xievito commented 1 year ago

I have the same problem. When I retrained the model, I couldn't reproduce the results of the paper. Also, the accuracy is very low! ! !

bruceyo commented 1 year ago

I think this is due to the skeleton data preparation issue (the MS-G3D's preparation can be different from those of 2s-AGCN and ST-GCN). For details, please refer to Issue 5.

xievito commented 1 year ago

I think this is due to the skeleton data preparation issue (the MS-G3D's preparation can be different from those of 2s-AGCN and ST-GCN). For details, please refer to Issue 5.

Thanks for your reply, I downloaded the MS-G3D dataset you provided, it is my train_rgb_fused.yaml configuration

# python main_rgb_fused.py recognition -c config/ntu60_xsub/train_rgb_fused.yaml
work_dir: /data/MMNet/work_dir/st-gcn/xsub/rgb_tp15_tf5_

skeleton_joints_pkl: /data/multi_model_data/results/ntu60/xsub/joint_result_msg3d.pkl
skeleton_bones_pkl: /data/multi_model_data/results/ntu60/xsub/bone_result_msg3d.pkl

# feeder
feeder: feeder.feeder_rgb_fused_ntu.Feeder
train_feeder_args:
  debug: False
  random_choose: False
  centralization: False
  random_move: False
  window_size: -1
  random_flip: False
  random_interval: True
  temporal_rgb_frames: 5
  data_path: /data/multi_model_data/data/ntu/xsub/train_data_joint.npy
  label_path: /data/multi_model_data/data/ntu/xsub/train_label.pkl
test_feeder_args:
  debug: False
  centralization: False
  evaluation: True
  temporal_rgb_frames: 5
  data_path: /data/multi_model_data/data/ntu/xsub/val_data_joint.npy
  label_path: /data/multi_model_data/data/ntu/xsub/val_label.pkl

# model
model: net.mmn.Model
model_args:
  in_channels: 3
  num_class: 60
  dropout: 0.5
  edge_importance_weighting: True
  graph_args:
    layout: 'ntu-rgb+d'
    strategy: 'spatial'

# training
temporal_positions: 15
fix_weights: True
joint_weights: /data/MMNet/work_dir/st-gcn/xsub/joint/epoch60_model.pt
device: [1,3]
weight_decay: 0.0001
base_lr: 0.01
step: [10, 50]
batch_size: 32
test_batch_size: 32
num_epoch: 80

# debug
debug: False

If convenient, please provide the corresponding yaml file. I would appreciate your prompt reply.

bruceyo / MMNet

Training RGB video (ntu60 xsub) #8