Create and train new dataset

[radhavishnu]

Hey! I want to create a new dataset having the following gestures start, stop, left, right. I have followed the instructions as mentioned by you Custom dataset creation

Plug in your realsense rgbd camera.

Run save_dataset.py to load the color and depth images from the realsense and store them.

python3 save_dataset.py --dataset-path dataset_images --gesture your_gesture

Place yourself in the camera FOV

Press "S" key to start the recording and "Q" key to stop it. Press "ctrl + C" to exit.

Repeat this step for all the gestures you want to record, considering that dataset_images is the output path for your dataset and your_gesture is the gesture you are saving

Annotate each of the gestures previously recorded:

python3 detect_skeleton.py --source dataset_images --view-img --annot

This command annotates all the previously recorded gestures. --view-img shows the images while detecting the skeleton. --annot annotates the gestures inside the folder in which the gesture is stored.

I have stored the images in /gestures3d/dataset_images/start/00001/color/ and /gestures3d/dataset_images/start/00001/depth/ respectively for ALL the actions

I am not able to annotate using the code

python3 detect_skeleton.py --source dataset_images --view-img --annot

It throws up the following error

root@21d1f6f9cd7c:/action3d# python3 detect_skeleton.py --source dataset_images --view-img --annot
Namespace(weights='weights/yolov7-w6-pose.pt', source='dataset_images', img_size=640, device='', view_img=True, annot=True, annot_file_name='annotations_3fps.json')
Writing skeleton for left gesture, example 1
YOLOR 🚀 2023-12-11 torch 1.13.1 CPU

Fusing layers... 
/opt/conda/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525552843/work/aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Model Summary: 494 layers, 80178356 parameters, 441076 gradients, 101.6 GFLOPS
Traceback (most recent call last):
  File "/action3d/detect_skeleton.py", line 173, in <module>
    detect(weights, view_img, imgsz, annot, os.path.join(source, c, record))
  File "/action3d/detect_skeleton.py", line 40, in detect
    sk = Skeleton3D(annot=annot, show_images=view_img, annot_name=annot_path)
  File "/action3d/utils/skeleton.py", line 101, in __init__
    pair = [(k, v) for k, v in annot_dict.items() if k in annot_name][0]
  File "/action3d/utils/skeleton.py", line 101, in <listcomp>
    pair = [(k, v) for k, v in annot_dict.items() if k in annot_name][0]
TypeError: 'in <string>' requires string as left operand, not int
Exception ignored in: <function Skeleton3D.__del__ at 0x7f8060792ef0>
Traceback (most recent call last):
  File "/action3d/utils/skeleton.py", line 363, in __del__
    if self.video_color is not None:
AttributeError: 'Skeleton3D' object has no attribute 'video_color'

[javierganan99]

Hi! Thank you for your interest. I have reproduced your error and you were right. I have corrected it, so you can clone the repo again and see if it works now. I no longer have a realsense camera :(, so I can not test it further. I would appreciate it if you could provide feedback that the other functionalities also work. If you are using the docker, you can share the volume of the project folder for these changes to take effect since I have not updated the docker image yet.

[radhavishnu]

Thank you for the prompt response! Yes i am using docker. The demo command using simple_images works perfectly well The following error still persists after running

python3 detect_skeleton.py --source dataset_images --view-img --annot

Namespace(weights='weights/yolov7-w6-pose.pt', source='dataset_images', img_size=640, device='', view_img=True, annot=True, annot_file_name='annotations_3fps.json')
Writing skeleton for left gesture, example 1
YOLOR 🚀 2023-12-11 torch 1.13.1 CPU

Fusing layers... 
/opt/conda/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525552843/work/aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Model Summary: 494 layers, 80178356 parameters, 441076 gradients, 101.6 GFLOPS
Traceback (most recent call last):
  File "/action3d/detect_skeleton.py", line 221, in <module>
    detect(
  File "/action3d/detect_skeleton.py", line 40, in detect
    sk = Skeleton3D(annot=annot, show_images=view_img, annot_name=annot_path)
  File "/action3d/utils/skeleton.py", line 106, in __init__
    pair = [(k, v) for k, v in annot_dict.items() if v in annot_name][0]
IndexError: list index out of range
Exception ignored in: <function Skeleton3D.__del__ at 0x7fa628bb2ef0>
Traceback (most recent call last):
  File "/action3d/utils/skeleton.py", line 404, in __del__
    if self.video_color is not None:
AttributeError: 'Skeleton3D' object has no attribute 'video_color'

Did u mean to share the entire folder or the dockerfile alone? I am attaching the folder containing the gestures3d directory

https://amritavishwavidyapeetham-my.sharepoint.com/:u:/g/personal/cb_en_u4aie20074_cb_students_amrita_edu/EVnhmwsva2FPtnUaMWBD7jUBjgiA0A8ZFq-paIA4DIHnNQ?e=JgxpnA

[javierganan99]

Hi @radhavishnu, thank you for your interest again, and sorry for the drawbacks. There is an error in the README, not in the python code. I should have specified an additional step to create your dataset, that is the first step of the model training section:

ST-GCN model training

In order to train yout custom ST-GCN model for gesture prediction:

1. Configure the following configuration files inside the cfg folder:

   • classes.yaml: Indicate the classes of your dataset.

I am going to correct it, thanks!

[radhavishnu]

Hey!! Thank you.... I have trained the model for my custom dataset. Its working absolutely fine. I can use the gesture_predict.py to evaluate my model after i have finished training? :)

[javierganan99]

Yes! Perfect, I hope it has been useful.