RuntimeError: Inconsistent object during array creation? Content of sequences changed (now too shallow)

[akamob]

Hi, Thank you for the awesome work.

I reproduce your work successfully and I use it to test with a webcam. I change the input_rgb_image topic to my topic (/webcam/image_raw) and follow your tutorial, but I got the following error:

Using GPU:  True
[INFO] [1644476763.510513]: Initialised ActivityRecognizer
[INFO] [1644476763.511202]: Ready to start...
[INFO] [1644476773.112062]: Got command to recognize activity
Traceback (most recent call last):
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 207, in <module>
    ar.run()
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 88, in run
    state = self.running_state()
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 122, in running_state
    clip = self.get_torch_clip(self.image_queue)
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 170, in get_torch_clip
    clip = np.asarray(frames, dtype=np.float32)
RuntimeError: Inconsistent object during array creation? Content of sequences changed (now too shallow).
[recognize_activity-1] process has died [pid 19817, exit code 1, cmd /home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity ~input_rgb_image:=/webcam/image_raw ~result:=/metrics_refbox_client/activity_recognition_result __name:=recognize_activity __log:=/home/lk3696/.ros/log/0675a816-8a34-11ec-b55c-0f78eddb9f48/recognize_activity-1.log].
log file: /home/lk3696/.ros/log/0675a816-8a34-11ec-b55c-0f78eddb9f48/recognize_activity-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done

Instead of using a live camera, I try to record my topic as a rosbag file and play it, but I still got same error.

May i have your suggestions to fix this problem? Any help is much appreciated:)

[sthoduka]

hi @akamob, this may be related to the version of numpy that you are using. I have been testing this with numpy version 1.17.4; which version are you using? You could try replacing this line with:

clip = np.array(frames)
clip = clip.astype('float32')

to see if it makes a difference.

[akamob]

Hi, @sthoduka Thank you very much for your reply:)

My numpy version is : 1.22.1 and after modifying recognize_activity with your suggestion, it works now:) But it can only runs for a few seconds (debug_image is not responding).

My GPU is GeForce GTX 1650 SUPER and when I run the code, the memory usage is about 50%. I'm not sure why debug_image is not responding.

Is there any way to to improve this problem? Any suggestions is much appreciated:)

[sthoduka]

Try to find out where it's getting stuck by placing rospy.loginfo statements in several places. You can also try to lower the parameters average_over_n_clips and result_n_clips.

[sthoduka]

Actually, with the default settings, it will only run for ~150 frames (around 3 seconds?). It is not set up to run continuously.

[akamob]

Hi, @sthoduka I am really thank you for quick response.

Now I can let debug_image response continuously, and recently I am preparing my custom data. These data come from a Velodyne lidar sensor (VLP-16), I convert point clouds into images like this:

These images are recorded as a rosbag file with topic /depth_image (Type: sensor_msgs/Image), I can transform it into mp4 format in order to fine-tune neural network. However, when I try to use your code to subscribe my data for activity recognition (not fine-tune yet), I met the following error:

Using GPU:  True
[INFO] [1645271487.506485]: Initialised ActivityRecognizer
[INFO] [1645271487.507241]: Ready to start...
[INFO] [1645271495.808131]: Got command to recognize activity
/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity:171: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  clip = np.array(frames)
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 209, in <module>
    ar.run()
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 88, in run
    state = self.running_state()
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 122, in running_state
    clip = self.get_torch_clip(self.image_queue)
  File "/home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity", line 172, in get_torch_clip
    clip = clip.astype('float32')
ValueError: setting an array element with a sequence.
[recognize_activity-1] process has died [pid 13642, exit code 1, cmd /home/lk3696/activity_recognition/src/activity_recognition_ros/scripts/recognize_activity ~input_rgb_image:=/depth_image ~result:=/metrics_refbox_client/activity_recognition_result __name:=recognize_activity __log:=/home/lk3696/.ros/log/18027ac2-9179-11ec-b3d4-eb0ee38e991d/recognize_activity-1.log].
log file: /home/lk3696/.ros/log/18027ac2-9179-11ec-b3d4-eb0ee38e991d/recognize_activity-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done

I write some codes to check these images:

import cv2
from PIL import Image

img_pil = Image.open('../depth_image/0037.png')
print('Pillow: ', img_pil.mode, img_pil.size)

img = cv2.imread('../depth_image/0037.png', cv2.IMREAD_UNCHANGED)
print('OpenCV: ', img.shape)

Output:

Pillow:  RGB (150, 64)
OpenCV:  (64, 150, 3)

I want to use a lidar for activity recognition, treat point clouds as RGB images. I'm wondering if it is possible to achieve it? and are there any limitations on the data? (such as image size, image channels, bit depth, etc.)

May i have your suggestions? Any help is much appreciated:)

[sthoduka]

yes, the code and the network expect 3 channel images as input. The image_transform also expects the image size to be at least 448 pixels in each dimension. You could change the parameters of the image_transform to match your image size. For the channels, one option would be to just duplicate the depth channel 3 times so that you end up with a 3-channel image.