Is this able to classify actions like sitting, standing, and lying down?

[Amyhds]

First, thank you for sharing this amazing work. I'm using Jetson TX2 and usb webcam. I'm wondering that this library automatically classifies normal actions like I mentioned without additional training. If it doesn't, does it only classify squatting and spinning? I look forward to your reply!

[smellslikeml]

Hi and thank you for your interest! We included a sample model under models/lstm_spin_squat.h5 to differentiate between those two actions for demo purposes. This repo should help you train your own model for your desired set of action categories following the guidance in the README section We were able to differentiate between a handful of movements very easily with a handful of samples as shown in the IVA example.

[Amyhds]

Hi and thank you for your interest! We included a sample model under models/lstm_spin_squat.h5 to differentiate between those two actions for demo purposes. This repo should help you train your own model for your desired set of action categories following the guidance in the README section We were able to differentiate between a handful of movements very easily with a handful of samples as shown in the IVA example.

I followed the steps but there's a issue. You said that the pickled model will be saved in the models/ directory after running train.py. Is the pickled model classifier.sav? So I changed the model and motion dict as my own model and classes but got this error.

nvidia@nvidia-desktop:~/ActionAI$ python iva.py 0
/usr/local/lib/python3.6/dist-packages/sklearn/utils/linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  DeprecationWarning)
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-amxqtk7m because the default path (/home/nvidia/.cache/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Matplotlib is building the font cache; this may take a moment.
[ WARN:0] global /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap_gstreamer.cpp (1761) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Device '/dev/video0' is busy
[ WARN:0] global /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap_gstreamer.cpp (888) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ WARN:0] global /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap_v4l.cpp (893) open VIDEOIO(V4L2:/dev/video0): can't open camera by index
[ERROR:0] global /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap.cpp (563) open VIDEOIO(GSTREAMER): raised OpenCV exception:

OpenCV(4.4.0) /tmp/pip-install-qd7trql4/opencv-contrib-python_04b7e37e30fe4a438c1485c3a4bc9161/opencv/modules/videoio/src/cap_gstreamer.cpp:1395: error: (-215:Assertion failed) frameSize.width > 0 && frameSize.height > 0 in function 'open'

2021-11-16 21:53:39.382516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Traceback (most recent call last):
  File "iva.py", line 256, in <module>
    secondary_model = tf.keras.models.load_model('models/classifier.sav')
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 149, in load_model
    loader_impl.parse_saved_model(filepath)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 83, in parse_saved_model
    constants.SAVED_MODEL_FILENAME_PB))
OSError: SavedModel file does not exist at: models/classifier.sav/{saved_model.pbtxt|saved_model.pb}

[Amyhds]

It succeeded when I ran with video file, but I still don't know why my webcam doesn't work.. It is not that accurate so I wonder if I should get more image samples. (only 8 pictures in each class..I couldn't use MPII because there's no enough memory)

[smellslikeml]

From the traceback, it looks like the device /dev/video0 is busy, meaning another process is using it. Perhaps you could look to see what process may be occupying the webcam or attach another webcam and try it (it might be read as /dev/video1 and so you'd pass a 1 to the iva.py script)

Some suggestions for improving accuracy:

• more training samples
• training samples that are representative of your test environment
• incorporate slight variation in training samples (distance, angles) for a more robust model

We are exploring major updates to the project. Feel free to join us on Gitter with your feedback or thoughts about improvements!

[Amyhds]

I'm really thankful for your help!!! 😆

[Amyhds]

sorry but I got problems again:(

I thought iva.py worked with mp4 using classifier.sav but actually lstm_spin_squat.h5 was used.

These are the result after running train.py

nvidia@nvidia-desktop:~/ActionAI$ python train.py
2021-11-24 21:03:24.418602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
INFO: Initialized TensorFlow Lite runtime.
/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
  "this warning.", FutureWarning)

and I got these errors running iva.py with classifier.sav.

nvidia@nvidia-desktop:~/ActionAI$ python iva.py test_vid_1.mp4
/usr/local/lib/python3.6/dist-packages/sklearn/utils/linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  DeprecationWarning)
2021-11-24 20:52:15.511297: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Traceback (most recent call last):
  File "iva.py", line 256, in <module>
    secondary_model = tf.keras.models.load_model('models/classifier.sav')
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 149, in load_model
    loader_impl.parse_saved_model(filepath)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 83, in parse_saved_model
    constants.SAVED_MODEL_FILENAME_PB))
OSError: SavedModel file does not exist at: models/classifier.sav/{saved_model.pbtxt|saved_model.pb}

what type of file I get after running train.py? I think it should be like "[modelname].h5" but there are 'classifier.sav, human_pose.json, lstm_spin_squat.h5, pose.tflite, resnet18_baseline_att_224x224_A_epoch_249_trt.pth' in models directory.

[Amyhds]

Hi @smellslikeml ! This time I modified PoseExtractor to TRTPoseExtractor in train.py and then got this error. What is the problem? Thank you!

~/ActionAI$ python train.py
2021-11-29 21:56:11.699266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
  "this warning.", FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/svm/base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  "the number of iterations.", ConvergenceWarning)

[smellslikeml]

These look like warnings. These questions are best to post on stack overflow where someone may have already answered it.

[Amyhds]

my current situation

• changed classifier_model = 'models/classifier.sav' to 'models/mymodel.h5' in conf.py.
• got the same problem 'OSError: SavedModel file does not exist at ...'
• tried 'sudo apt install python3-h5py' but nothing changed
• still finding a solution

[LawrenceXu13467]

Hi Amyhds,

Following your previous conversation, have you figured out how to solve these problems? I got into a similar situation here. Thank you.

Sincerely, Lawrence

[Amyhds]

Hi @LawrenceXu13467 I could not find solutions..sorry