Full ek55 test set results

[richwardle]

How do I implement your model on the full EK55 test set? I am aiming to gain the results shown in Table 3 in your paper. I have all the datasets and csv as you spoke about in the readme. Thanks!

[debadityaroy]

Hi @r3ward

• You need to train two models - one using TSNRGB features and another using bag of object (OBJ) features obtained from RULSTM repository. Training hyper params - obs_segs = 1, seg_length_secs = 2, epochs = 10

• For EK55 test set, use the evaluation code which combines verb predictions from TSNRGB model and noun predictions from OBJ model.

[richwardle]

Thanks! May I have an email address to reach you on? Have a few more questions about the implimentation

[debadityaroy]

You can contact me on this address.

On Fri, 20 May, 2022, 00:16 Richard Wardle, @.***> wrote:

Thanks! May I have an email address to reach you on? Have a few more questions about the implimentation

— Reply to this email directly, view it on GitHub https://github.com/debadityaroy/LatentGoal/issues/2#issuecomment-1131920974, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUBUXB6HUFFWQAIMP7IWY3VKZSNDANCNFSM5VMIGABA . You are receiving this because you commented.Message ID: @.***>

[richwardle]

• I trained both RGB and OBJ with the mentioned dimensions using tsnrgb_feat_latent_goal_action_max_current_action.py
• This put two separate .pt files into /ckp, I then placed all the parameters into latent_goal_inference_rgb+obj.py.

Just to mention a change i had to make: Lines 103 in latent_goal_inference_rgb+obj.py created a depreciated tensor error (only from the rgb feature) '.copy() - I switching it to '.clone()'.

When I run the program, I am getting a parser error when reading in 'seen.csv' and 'unseen.csv' into the evaluate function. The reading of the csv is throwing the error - 'expecting 1 fields in line 2, saw 2'. Any idea of how to circumvent this? I am attempting to get validation results with different anticipation times and the test results. Thanks!

[debadityaroy]

1. I am not able to replicate your issue, I am able to parse both test_seen.csv and test_unseen.csv using the evaluate function. Are you using the csv files from https://github.com/fpv-iplab/rulstm/tree/master/RULSTM/data/ek55?

2. For different anticipation times, you need to add ant_time = ant_sec*30 where ant_sec is the anticipation seconds (default 1.0s). Then change stop = stop - ant_time https://github.com/debadityaroy/LatentGoal/blob/cf49bc7b41f2f66cddcbdbca5dd3939e325cc084/latent_goal_inference_rgb%2Bobj.py#L89, commented code is already present.

[richwardle]

Thanks for the response, I'm retraining for obj and rgb in case I had an incorrect parameter somewhere - I'll let you know how I get on. Did you ever get results in Top-5 Action Accuracy for timesteps between 0.25 seconds and 2.0 seconds? I noticed a couple of models review this as part of their findings, so including it within my survey and would appreciate it if you have these figures. Additionally, how did you extract findings from the seen and unseen JSONs, I have a seen.json but trying to find a script to parse it for the final results. Thanks!

[debadityaroy]

The results on different anticipation times are reported on validation set, not test set.

[richwardle]

Right, I'm trying to get all the Top-5 Action accuracies ranging from 0.25 to 2.0 seconds on the validation set. Did you ever get these results? Collecting the results like this: https://ibb.co/z62snVQ

[debadityaroy]

I can't access your link.