n_frames

[olgapeled]

Hi,

I am trying to use your AMAZING code with the jester dataset. I have some questions:

1. I see that you assume that for each video there is a directory names "n_frames". How can I create those directories? I downloaded the jester dataset and extracted the data as describes in the link but there aren't directories with the name "n_frames"
2. Can you give the parameters for testing on the jester dataset?

Thank you!

[olgapeled]

I think that jester_prepare in utils is missing

[ahmetgunduz]

Hi,

1. As far as I remember, for jester you can use utils/jester_json.py to create the json object. But you need to provide the path of annotation_Jester as csv_dir_path. Can you try python ./utils/jester_json.py "./annotation_Jester"?

This should create a json object. Then when you are running the code, just provide path of jester dataset as --video_path

2. You must specify the number of classes as 27 and the rest is the same as the others.

Hope this is clear.

[ahmetgunduz]

Hi @olgapeled, If that worked for you, please let me know. Otherwise, I am closing the issue?

[olgapeled]

Hi,

1. As far as I remember, for jester you can use utils/jester_json.py to create the json object. But you need to provide the path of annotation_Jester as csv_dir_path. Can you try python ./utils/jester_json.py "./annotation_Jester"?

This should create a json object. Then when you are running the code, just provide path of jester dataset as --video_path

1. You must specify the number of classes as 27 and the rest is the same as the others.

Hope this is clear.

Jester dataset consists of 148,093 directories in which each directory contains a set of frames that represent a video. I already created the json object. When I am trying to run the code, I provide the jester dataset as --video_path which point to the main folder I have mentioned before, I got the next error: "FileNotFoundError: [Errno 2] No such file or directory: '/work/datasets/20BN/20bn-jester-v1/32122/n_frames'

After looking into the code jester.py lines: 114-115,
"n_frames_file_path = os.path.join(video_path, 'n_frames') n_frames = int(load_value_file(n_frames_file_path))" It seems that video_path is concatenated with a string "n_frames" and that this directory should contain some data. How can I create such structure of that dataset???

Thank you, Olga

[ahmetgunduz]

I see, usually there should be a default n_frames file in each folder. Anyway, you can simply count number of .jpg images in each folder like following .

import glob 
import os

# n_frames_file_path = os.path.join(video_path, 'n_frames')
# n_frames = int(load_value_file(n_frames_file_path))
n_frames = len(glob.glob(os.path.join(video_path, '*.jpg')))

[olgapeled]

I moved to the NV dataset just to make the code run. Now I am facing another problem. All the links in annotation_nvGesture/vallistall.txt end with "sk_color_all", but those links actually do not exist in the nv dataset structure. I assume that "sk_color_all" should be a directory with all the video frames ???

Each subject in each class in the NV dataset contains ONLY the next files: Video_data/class_01/subject1_r0/duo_left.avi Video_data/class_01/subject1_r0/duo_left_log.txt Video_data/class_01/subject1_r0/duo_right.avi Video_data/class_01/subject1_r0/duo_right_log.txt Video_data/class_01/subject1_r0/nucleus.txt Video_data/class_01/subject1_r0/sk_color.avi Video_data/class_01/subject1_r0/sk_color_log.txt Video_data/class_01/subject1_r0/sk_depth.avi Video_data/class_01/subject1_r0/sk_depth_log.txt Video_data/class_01/subject1_r0/sk_wrapped.avi Video_data/class_01/subject1_r0/sk_wrapped_log.txt

Can you please help me? Should I create those sk_color_all directories?? From which avi file?

Thank very much!!!!

[ahmetgunduz]

Hi, You are right! This is sth I should have included in the repo. Sorry for the confusion.

First, you need to create two sub folders for RGB and Depth videos, seperately. For your example above it would be like this Video_data/class_01/subject1_r0/sk_color_all or Video_data/class_01/subject1_r0/sk_depth_all

Then, all frames from the avi files with the same name need to be extracted into these folders. The code should be sth like below. Please make sure that you have ffmpeg installed if it is not already. Instructions in this site should be enough.

Unfortunately, I do not have the access to the dataset anymore so I cannot test the code below but it should work.

from subprocess import call
import glob
import os

video_types = ["color", "depth"]
for vt in video_types:
    files = glob.glob(os.path.join("Video_data", "*", "*", "sk_" + vt + ".avi")) # this line should be updated according to the full path 
    for file in files:
        directory = file.split(".")[0] + "_all"
        if not os.path.exists(directory):
            os.makedirs(directory)
        call(["ffmpeg", "-i",  file, os.path.join(directory, "%05d.jpg"), "-hide_banner"]) # it should be 5 digit number because in nv.py it is assumed to be like so

Please let me know if that works. I will update the repo accordingly if so .

[olgapeled]

Great!! It works! Do you have this also for the jester dataset? Can you please provide the structure you assume for the jester dataset?

More questions:

I am trying to run offline_test.py on the nv dataset.

1. Is it ok that I used the nvall_but_None.json for the offline test? because if I am using the nvall.json I got much lower results...
2. It seems that your pre-trained models are in depth modality but, in the jsons files and in the vallists all the paths are of the colors videos and not the depth ones. What I am missing?
3. Do you have pre-trained models in the "RGB" modality?

Thank you very much!!!!!!!!!

[ahmetgunduz]

Hi olgapeled, I used the default structure path of jester as following 20bn-jester-v1/32122/<%05d.jpg> For Jester you o not need because it has only rgb images and the /datasets/jester.py should handle this.

1. nvall_but_None would be ok but I definitelly recommend to use pretrained models on Jestera ss initialization otherwise nvidia dataset is too small to train a ResNeXt-101 for example. Also the accuracy of your model is low in case you use nvall.json because half of the dataset is NO_GESTURE class.

2. When the modality of the pretrained model and the model you want to train is not the same I change the first layer. See lines 70-80 in model.py :

   
       if opt.modality == 'RGB' and opt.model != 'c3d':
           print("[INFO]: RGB model is used for init model")
           model = _modify_first_conv_layer(model,3,3) ##### Check models trained (3,7,7) or (7,7,7)
       elif opt.modality == 'Depth':
           print("[INFO]: Converting the pretrained model to Depth init model")
           model = _construct_depth_model(model)
           print("[INFO]: Done. Flow model ready.")
       elif opt.modality == 'RGB-D':
           print("[INFO]: Converting the pretrained model to RGB+D init model")
           model = _construct_rgbdepth_model(model)
           print("[INFO]: Done. RGB-D model ready.")

3. Only for ResNeXt-101 with Jester. Please check the drive link in README.

I am closing the issue as n_frames is solved. Please open a new one in case of a problem.

[Nd-sole]

Hey @ahmetgunduz and @olgapeled

I am also getting same error which @olgapeled got. What is n_frames in the dataset (the string in the name of data path) FileNotFoundError: [Errno 2] No such file or directory: '/media/storage/ndhingra/Jester/datatest/20bn-jester-v1/63423/n_frames'

@olgapeled, if you solved this problem, could you tell me?

@ahmetgunduz, could you explain what are you referring to here n_frames_file_path = os.path.join(video_path, 'n_frames') this line is taken from Jester.py from function/method named "make_dataset"

Thanks :)

[ahmetgunduz]

Hi @nainadhingra2012 , I had updated the code with this commit 55eb514. Can you please pull the repo again? Best, Ahmet

[Nd-sole]

I moved to the NV dataset just to make the code run. Now I am facing another problem. All the links in annotation_nvGesture/vallistall.txt end with "sk_color_all", but those links actually do not exist in the nv dataset structure. I assume that "sk_color_all" should be a directory with all the video frames ???

Each subject in each class in the NV dataset contains ONLY the next files: Video_data/class_01/subject1_r0/duo_left.avi Video_data/class_01/subject1_r0/duo_left_log.txt Video_data/class_01/subject1_r0/duo_right.avi Video_data/class_01/subject1_r0/duo_right_log.txt Video_data/class_01/subject1_r0/nucleus.txt Video_data/class_01/subject1_r0/sk_color.avi Video_data/class_01/subject1_r0/sk_color_log.txt Video_data/class_01/subject1_r0/sk_depth.avi Video_data/class_01/subject1_r0/sk_depth_log.txt Video_data/class_01/subject1_r0/sk_wrapped.avi Video_data/class_01/subject1_r0/sk_wrapped_log.txt

Can you please help me? Should I create those sk_color_all directories?? From which avi file?

Thank very much!!!!

Hi @olgapeled and @ahmetgunduz When I download the dataset fron G drive by nvidia. The data is in the form: /nvgesture_arch-20190604T091542Z-013/nvgesture_arch/nvGesture_v1.7z.003 /nvgesture_arch-20190604T091542Z-013/nvgesture_arch/nvGesture_v1.7z.004 /nvgesture_arch-20190604T091542Z-013/nvgesture_arch/nvGesture_v1.7z.005 . . . . /nvgesture_arch-20190604T091542Z-013/nvgesture_arch/nvGesture_v1.7z.031

How do you convert/process the data so that the nv_prepare can be used?