Kinetics results do not match with the paper

[0AndyH0]

Hi Chiara,

Really enjoy your paper. Thank you for sharing the code as well. We (@sneheshs and I) were trying to reproduce the results and only got 0.25% for the top-1 instead of 37.0% as reported in the paper. The only thing we had to change was the number of channels from 6 to 3 and double channel to False as the Kinetics dataset does not have bones which perhaps is causing this discrepancy.

  channel: 3 # 6
  double_channel: False # True

Based on the train.yaml file, however, we noticed that you are using 6 channels with bones data which the currently published script kinetics_gendata.py does not output. Could you please share the updated file so we can reproduce the results accurately?

Training config file: https://github.com/Chiaraplizz/ST-TR/blob/master/code/config/st_gcn/kinetics-skeleton/train.yaml

test_feeder_args:
  data_path: ./kinetics_data/val_data_joint_bones.npy

Thanks for your help.

Best, Snehesh and Andy

[sneheshs]

Hi Chiara,

We noticed that in https://github.com/Chiaraplizz/ST-TR/blob/master/code/tools/ntu_gen_bones.py

# datasets = {'kinetics'} if kinetics is used
datasets = {'xsub', 'xview'}

We weren't sure if this is the right way to generate the bones data for kinects but that was the only way we noticed, so we generated kinetics bones by changing the datasets value to kinetics. And we used --weights='checkpoint_ST-TR/kinetics_bones_temporal.pt' as the pretrained model weights.

However, the results for kinetics bone kinetics 6 channel seems very low:

Top1: 2.62%
Top5: 9.16% 

For kinects 3 channel, we are getting consistent with paper of:

Top1: 32.42% 
Top5: 55.23% 

So we think the way we generated bones is perhaps not correct. Can you please advise the correct way to generate the bones data?

Best, Andy and Snehesh

[Chiaraplizz]

Hi Chiara,

We noticed that in https://github.com/Chiaraplizz/ST-TR/blob/master/code/tools/ntu_gen_bones.py

# datasets = {'kinetics'} if kinetics is used
datasets = {'xsub', 'xview'}

We weren't sure if this is the right way to generate the bones data for kinects but that was the only way we noticed, so we generated kinetics bones by changing the datasets value to kinetics. And we used --weights='checkpoint_ST-TR/kinetics_bones_temporal.pt' as the pretrained model weights.

However, the results for kinetics bone kinetics 6 channel seems very low:

Top1: 2.62%
Top5: 9.16% 

For kinects 3 channel, we are getting consistent with paper of:

Top1: 32.42% 
Top5: 55.23% 

So we think the way we generated bones is perhaps not correct. Can you please advise the correct way to generate the bones data?

Best, Andy and Snehesh

Dear Snehesh and Andy,

The function you used to generate them is correct, but in order to obtain 6 channels you should concatenate bones information to joint information (3 channels * 2). You can do it through the script ntu_merge_joint_bones.py.

Chiara :)

[sneheshs]

Dear Snehesh and Andy,

The function you used to generate them is correct, but in order to obtain 6 channels you should concatenate bones information to joint information (3 channels * 2). You can do it through the script ntu_merge_joint_bones.py.

Chiara :)

Thanks for getting back to us Chiara. Sorry, we should have mentioned that we did run the merge step as well. For some reason, the 6 channel results are not working for us. Is there anything we might have missed in our steps?

[Chiaraplizz]

Dear Snehesh and Andy, The function you used to generate them is correct, but in order to obtain 6 channels you should concatenate bones information to joint information (3 channels * 2). You can do it through the script ntu_merge_joint_bones.py. Chiara :)

Thanks for getting back to us Chiara. Sorry, we should have mentioned that we did run the merge step as well. For some reason, the 6 channel results are not working for us. Is there anything we might have missed in our steps?

Can you share your train.yaml?

[sneheshs]

Sure, please see attached. Thank you! train.yaml.zip

[Chiaraplizz]

Sure, please see attached. Thank you! train.yaml.zip

This seems correct! Does it throw any error when you load weights?

Did you try to run the bones generation again? It happened also to me once that it interrupted during generation and the bones of some samples were not complete (you can simply verify if there are some samples where the bones values are always zero). I suggest you to verify that, and eventually try to run again the generation. Let me know!

Maybe if you can please share also the main.py I can also give a look to that.

Chiara

[sneheshs]

Sure, please see attached. Thank you! train.yaml.zip

This seems correct! Does it throw any error when you load weights?

Did you try to run the bones generation again? It happened also to me once that it interrupted during generation and the bones of some samples were not complete (you can simply verify if there are some samples where the bones values are always zero). I suggest you to verify that, and eventually try to run again the generation. Let me know!

Maybe if you can please share also the main.py I can also give a look to that.

Chiara

No, it doesn't throw any errors. It just shows the tensor size as it runs as expected and spits out the results. We can definitely checkout the bones values to see if there are any large sets of all-zeros values -- regardless we will try to regenerate and run again like you suggested. In the mean time, here is the main.pay. Looking at some to paths, I am wondering if I made a mistake any one of the paths.

main.py.zip

[sneheshs]

BTW we checked the bones data and we do see a lot of entries with zeros. We are not sure if this is naturally expected zeros or "bad" ones. Would it be possible for you to share the kinetics bones data for us to compare with ours?

[Chiaraplizz]

BTW we checked the bones data and we do see a lot of entries with zeros. We are not sure if this is naturally expected zeros or "bad" ones. Would it be possible for you to share the kinetics bones data for us to compare with ours?

I might share them with you, but I think it is easier for you to write a simple script that checks if your data_bones[:, :, :, v1, :] == (data_joint[:, :, :, v1, :] - data_joint[:, :, :, v2, :]) as it has supposed to be calculated in ntu_gen_bones.py.

Send me an e-mail you have troubles, and I eventually will share the data with you.

Chiara

[sneheshs]

Hi Chiara, some good news. We re-ran the code and regenerated the bones data. We found that there were some error messages on our part, and after fixing, we are getting results in line with the paper of top1: 33.09% and top5: 55.86%. Thank you so much for feedback and help.

[Chiaraplizz]

Hi Chiara, some good news. We re-ran the code and regenerated the bones data. We found that there were some error messages on our part, and after fixing, we are getting results in line with the paper of top1: 33.09% and top5: 55.86%. Thank you so much for feedback and help.

You're welcome :)