An unofficial PyTorch implementation of the paper "Skeleton-Based Action Recognition with Directed Graph Neural Networks" in CVPR 2019.
NOTE: Experiment results are not being updated due to hardware limits.
Most of the interesting stuff can be found in:
model/dgnn.py
: model definition of DGNNdata_gen/
: how raw datasets are processed into numpy tensorsgraphs/directed_ntu_rgb_d.py
: graph definition for DGNNfeeders/feeder.py
: how datasets are read inmain.py
: general training/eval processes; graph freezing by disabling gradients; etc.The NTU RGB+D dataset can be downloaded from here. We'll only need the Skeleton data (~ 5.8G).
After downloading, unzip it and put the folder nturgb+d_skeletons
to ./data/nturgbd_raw/
.
Generate the joint dataset first:
cd data_gen
python3 ntu_gen_joint_data.py
Specify the data location if the raw skeletons data are placed somewhere else. The default looks at ./data/nturgbd_raw/
.
data_gen/
, generate the bone dataset:python3 ntu_gen_bone_data.py
python3 ntu_gen_motion_data.py
The generation scripts look for generated data in previous step. By default they look at ./data
; change dir configs if needed.
(Currently, generating bone/motion data from Kinetics skeletons is not yet supported. Please feel free to add scripts based on kinetics_gendata.py
)
cd data_gen
python3 kinetics_gendata.py
To start training the network with the spatial stream, use the following command:
python3 main.py --config ./config/<dataset>/train_spatial.yaml
Here, <dataset>
should be one of nturgbd-cross-subject
, nturgbd-cross-view
, or kinetics-skeleton
depending on the dataset/task on which to train the model.
Note: At the moment, only nturgbd-cross-subject
is supported. More config files will (hopefully) be added, or you could write your own config file using the existing ones for nturgbd-cross-subject
.
Similarly, to train on the motion stream data, do:
python3 main.py --config ./config/nturgbd-cross-subject/train_motion.yaml
and change the config file path for other datasets if needed.
To test some model weights (by default saved in ./runs/
), do:
python3 main.py --config ./config/<dataset>/test_spatial.yaml
Similarly, change the paths in config file, or change the config files (<dataset>
) for different datasets as needed.
Combine the generated scores with:
python ensemble.py --datasets <dataset>
where <dataset>
is one of kinetics
, ntu/xsub
, ntu/xview