search

smellslikeml / ActionAI

Real-Time Spatio-Temporally Localized Activity Detection by Tracking Body Keypoints
https://www.hackster.io/actionai/actionai-custom-tracking-multiperson-activity-recognition-fa5cb5
GNU General Public License v3.0
772 stars 192 forks source link

TypeError: tuple indices must be integers or slices, not tuple #48

Open hamza90ec opened 3 years ago

hamza90ec commented 3 years ago

I am using jetson nano tx1. can you please debug this error here is my full terminal status [TensorRT] WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors. [ WARN:0] global /home/hamza/opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1 [ WARN:0] global /home/hamza/opencv/modules/videoio/src/cap_gstreamer.cpp (1184) setProperty OpenCV | GStreamer warning: GStreamer: unhandled property 2021-07-03 18:43:26.535555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-07-03 18:43:42.980931: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-07-03 18:43:43.216263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:43.252280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties: pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3 coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s 2021-07-03 18:43:43.252747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-07-03 18:43:43.266841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2021-07-03 18:43:43.267047: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-07-03 18:43:43.267235: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-07-03 18:43:43.267471: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-07-03 18:43:43.267641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2021-07-03 18:43:43.277523: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-07-03 18:43:43.277755: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:43.277983: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:43.292821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0 2021-07-03 18:43:44.054252: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency 2021-07-03 18:43:44.090574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ae52e50 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-07-03 18:43:44.090673: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-07-03 18:43:44.202240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:44.202702: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ae71050 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-07-03 18:43:44.202767: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3 2021-07-03 18:43:44.242034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:44.242247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties: pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3 coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s 2021-07-03 18:43:44.242830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-07-03 18:43:44.242999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2021-07-03 18:43:44.243104: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-07-03 18:43:44.243232: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-07-03 18:43:44.243363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-07-03 18:43:44.243472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2021-07-03 18:43:44.243576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-07-03 18:43:44.243784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:44.244020: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:43:44.244120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0 2021-07-03 18:44:09.863146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1283] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-03 18:44:10.010804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1289] 0 2021-07-03 18:44:10.010977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1302] 0: N 2021-07-03 18:44:10.386921: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:44:10.505899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero 2021-07-03 18:44:10.606799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 92 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU Traceback (most recent call last): File "iva.py", line 275, in matched, unmatched_trackers, unmatched_detections = tracker_match(track_boxes, [b[0] for b in bboxes]) File "iva.py", line 129, in tracker_match if(d not in matched_idx[:,1]): TypeError: tuple indices must be integers or slices, not tuple

LewsTherin511 commented 3 years ago

Having the same issue

JJLimmm commented 2 years ago

has anybody solved this yet?

Hi @smellslikeml do you have an idea what might have caused this?

smellslikeml commented 2 years ago

We haven't reproduced this one but suggest printing the matched_idx array just before line 129. We'll try updating the tracker matching logic

JJLimmm commented 2 years ago

Hi @smellslikeml , just some background info, i am running the iva.py script on my laptop, on a video of a single person doing squats repeatedly for 3 reps. The model i used is the original _resnet_baseline_att_224x224_A_epoch_249_trt.pth_ and a trained classifier.sav with images of a single person doing squats and standing (2 classes). I modified the code a little (changes shown below) to load the classifier.sav instead of the LSTM model which was by default in the script. 

if RUNSECONDARY:  
    import tensorflow as tf
    import pickle
    import importlib
    config = importlib.import_module('config.conf')
    # secondary_model = tf.keras.models.load_model('models/lstm_spin_squat.h5')
    secondary_model = pickle.load(open(config.classifier_model, 'rb'))
    window = 1
    pose_vec_dim = 36
    motion_dict = {0: 'standing', 1: 'squats'}

iva_py For the issue mentioned by @hamza90ec and also the screenshot i uploaded above, i believe it has something that has got to do with the first frame for any video/live feed. As you can see from the screenshot that even though there is a detection of 1 person in the frame (which is correct, only 1 person in the frame), but the tracker did not take into account this first frame not added to the "trackers" list.

I made some changes to the iva.py code for a quick fix around this but i still faced an error when the code runs up till extracting unmatched trackers and detections after the _linear_sumassignment from _IOUmat. The screenshot below shows the error after making the changes, and as you can see the first frame objects detected are initialized as tracks and appended to trackers for the next iteration to continue tracking.

iva_edited

The code below is what i have changed in iva.py to get this error.

iva_modified.txt Summary of changes to code: line 297 - 304: Added a flag for the first frame detections line 134 - 137: Some print statements for macthed_idx (output from _linearassignment() of _IOUmat) line 16: changed import of from sklearn.utils.linear_assignment import linear_assignment to from scipy.optimize import linear_sum_assignment as linear_assignment as the sklearn library is deprecated.

I was reading through the documentation for _scipy.optimize.linear_sumassignment(), the output is the row and column indexes for the minimum cost for IOU_mat. However, it is combined into one by stating the variable matched_idx. I don't quite understand why you did this. I added in some commented statements to print out the row and col indexes from the output of the _linearassignment() just to visualize the output.

Do you have an idea of what might be wrong with the code/or what we did wrong? @smellslikeml

Thanks!

smellslikeml commented 2 years ago

That's right, using linear_sum_assigment, you can return row indices and column indices for matches.

After the scipy matching, you want to add indices for unmatched tracker sfrom the previous timestep to a list. Something like: unmatched_trackers = list(set(range(len(trackers))).difference(rowIdxs))

Likewise, you can add indices of unmatched detections from the current timestep into a list. unmatched_detections = list(set(range(len(detections))).difference(colIdxs))

Later, we use these lists to filter tracklets we have lost or to initialize a new tracklet.

I will do some testing and update soon.

Thanks!

On Tue, Sep 27, 2022 at 6:56 PM JJ Lim | Eugene @.***> wrote:

Hi @smellslikeml https://github.com/smellslikeml , just some background info, i am running the iva.py script on my laptop, on a video of a single person doing squats repeatedly for 3 reps. The model i used is the original resnet_baseline_att_224x224_A_epoch_249_trt.pth and a trained classifier.sav with images of a single person doing squats and standing (2 classes). I modified the code a little (changes shown below) to load the classifier.sav instead of the LSTM model which was by default in the script.

if RUNSECONDARY: import tensorflow as tf import pickle import importlib config = importlib.import_module('config.conf')

secondary_model = tf.keras.models.load_model('models/lstm_spin_squat.h5')

secondary_model = pickle.load(open(config.classifier_model, 'rb'))
window = 1
pose_vec_dim = 36
motion_dict = {0: 'standing', 1: 'squats'}

[image: iva_py] https://user-images.githubusercontent.com/79782049/192667513-b5abccc7-3685-4c4b-8df1-b8c7b5007abe.png For the issue mentioned by @hamza90ec https://github.com/hamza90ec and also the screenshot i uploaded above, i believe it has something that has got to do with the first frame for any video/live feed. As you can see from the screenshot that even though there is a detection of 1 person in the frame (which is correct, only 1 person in the frame), but the tracker did not take into account this first frame not added to the "trackers" list.

I made some changes to the iva.py code for a quick fix around this but i still faced an error when the code runs up till extracting unmatched trackers and detections after the linear_sum_assignment from IOU_mat. The screenshot below shows the error after making the changes, and as you can see the first frame objects detected are initialized as tracks and appended to trackers for the next iteration to continue tracking.

[image: iva_edited] https://user-images.githubusercontent.com/79782049/192668482-75121eac-8745-430c-9816-72fb11fa3ba9.png

The code below is what i have changed in iva.py to get this error.

iva_modified.txt https://github.com/smellslikeml/ActionAI/files/9660719/iva_modified.txt Summary of changes to code: line 297 - 304: Added a flag for the first frame detections line 134 - 137: Some print statements for macthed_idx (output from linear_assignment() of IOU_mat) line 16: changed import of from sklearn.utils.linear_assignment import linear_assignment to from scipy.optimize import linear_sum_assignment as linear_assignment as the sklearn library is deprecated.

I was reading through the documentation for scipy.optimize.linear_sum_assignment(), the output is the row and column indexes for the minimum cost for IOU_mat. However, it is combined into one by stating the variable matched_idx. I don't quite understand why you did this. I added in some commented statements to print out the row and col indexes from the output of the linear_assignment() just to visualize the output.

Do you have an idea of what might be wrong with the code/or what we did wrong? @smellslikeml https://github.com/smellslikeml

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/smellslikeml/ActionAI/issues/48#issuecomment-1260289731, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFAHKY2EVWENIQNBOU3TKDWAOQTPANCNFSM47YFGQNA . You are receiving this because you were mentioned.Message ID: @.***>

JJLimmm commented 2 years ago

I see, alright i understand now. Thanks and hope to hear from you soon. I will continue to try maybe debug this as well, if i find a fix will post it here!

Cheers!