search

ros-ai / ros2_whisper

Whisper C++ Inference Action Server for ROS 2
29 stars 9 forks source link

Unknown PCM cards and ERROR not all tensors loaded from model file - expected 245, got 3 #11

Open robotx10 opened 10 months ago

robotx10 commented 10 months ago

admin@ubuntu:/workspaces/isaac_ros-dev$ ros2 launch whisper_bringup bringup.launch.py n_thread:=4 [INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-11-27-14-33-17-107142-ubuntu-172426 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [audio_listener-1]: process started with pid [172443] [INFO] [component_container_mt-2]: process started with pid [172445] [component_container_mt-2] [INFO] [1701124397.745416739] [whisper_container]: Load Library: /workspaces/isaac_ros-dev/install/whisper_server/lib/libinference_component.so [component_container_mt-2] [INFO] [1701124397.757107988] [whisper_container]: Found class: rclcpp_components::NodeFactoryTemplate [component_container_mt-2] [INFO] [1701124397.757246584] [whisper_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate [component_container_mt-2] Path already exists: /home/admin/.cache/whisper.cpp [component_container_mt-2] [INFO] [1701124397.790329612] [whisper.inference]: Checking whether model base.en is available... [component_container_mt-2] [INFO] [1701124397.790815291] [whisper.inference]: Model base.en is available. [component_container_mt-2] [INFO] [1701124397.790848477] [whisper.inference]: Initializing model base.en... [component_container_mt-2] whisper_init_from_file_with_params_no_state: loading model from '/home/admin/.cache/whisper.cpp/ggml-base.en.bin' [component_container_mt-2] whisper_model_load: loading model [component_container_mt-2] whisper_model_load: n_vocab = 51864 [component_container_mt-2] whisper_model_load: n_audio_ctx = 1500 [component_container_mt-2] whisper_model_load: n_audio_state = 512 [component_container_mt-2] whisper_model_load: n_audio_head = 8 [component_container_mt-2] whisper_model_load: n_audio_layer = 6 [component_container_mt-2] whisper_model_load: n_text_ctx = 448 [component_container_mt-2] whisper_model_load: n_text_state = 512 [component_container_mt-2] whisper_model_load: n_text_head = 8 [component_container_mt-2] whisper_model_load: n_text_layer = 6 [component_container_mt-2] whisper_model_load: n_mels = 80 [component_container_mt-2] whisper_model_load: ftype = 1 [component_container_mt-2] whisper_model_load: qntvr = 0 [component_container_mt-2] whisper_model_load: type = 2 (base) [component_container_mt-2] whisper_model_load: adding 1607 extra tokens [component_container_mt-2] whisper_model_load: n_langs = 99 [component_container_mt-2] ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no [component_container_mt-2] ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes [component_container_mt-2] ggml_init_cublas: found 1 CUDA devices: [component_container_mt-2] Device 0: Orin, compute capability 8.7 [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline [audio_listener-1] ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline [audio_listener-1] ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port [audio_listener-1] ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port [audio_listener-1] ALSA lib pulse.c:242:(pulse_connect) PulseAudio: Unable to connect: Connection refused [audio_listener-1] [audio_listener-1] ALSA lib pulse.c:242:(pulse_connect) PulseAudio: Unable to connect: Connection refused [audio_listener-1] [audio_listener-1] ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card [audio_listener-1] ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card [component_container_mt-2] whisper_model_load: CPU buffer size = 156.67 MB [component_container_mt-2] whisper_model_load: model size = 57.10 MB [component_container_mt-2] whisper_model_load: ERROR not all tensors loaded from model file - expected 245, got 3 [component_container_mt-2] whisper_init_with_params_no_state: failed to load model [component_container_mt-2] [INFO] [1701124401.647853668] [whisper.inference]: Model base.en initialized. [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/whisper/inference' in container '/whisper_container' ^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)

[ERROR] [audio_listener-1]: process has died [pid 172443, exit code -2, cmd '/workspaces/isaac_ros-dev/install/audio_listener/lib/audio_listener/audio_listener --ros-args']. [audio_listener-1] Traceback (most recent call last): [audio_listener-1] File "/workspaces/isaac_ros-dev/install/audio_listener/lib/audio_listener/audio_listener", line 33, in [audio_listener-1] sys.exit(load_entry_point('audio-listener', 'console_scripts', 'audio_listener')()) [audio_listener-1] File "/workspaces/isaac_ros-dev/build/audio_listener/audio_listener/audio_listener.py", line 70, in main [audio_listener-1] rclpy.spin(audio_listener) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/init.py", line 222, in spin [audio_listener-1] executor.spin_once() [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 705, in spin_once [audio_listener-1] handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 691, in wait_for_ready_callbacks [audio_listener-1] return next(self._cb_iter) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 588, in _wait_for_ready_callbacks [audio_listener-1] wait_set.wait(timeout_nsec) [audio_listener-1] KeyboardInterrupt [INFO] [component_container_mt-2]: process has finished cleanly [pid 172445]

robotx10 commented 10 months ago

[audio_listener-1] Traceback (most recent call last): [audio_listener-1] File "/workspaces/isaac_ros-dev/install/audio_listener/lib/audio_listener/audio_listener", line 33, in [audio_listener-1] sys.exit(load_entry_point('audio-listener', 'console_scripts', 'audio_listener')()) [audio_listener-1] File "/workspaces/isaac_ros-dev/build/audio_listener/audio_listener/audio_listener.py", line 73, in main [audio_listener-1] rclpy.spin(audio_listener) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/init.py", line 222, in spin [audio_listener-1] executor.spin_once() [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 713, in spin_once [audio_listener-1] raise handler.exception() [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/task.py", line 239, in call [audio_listener-1] self._handler.send(None) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 418, in handler [audio_listener-1] await call_coroutine(entity, arg) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 332, in _execute_timer [audio_listener-1] await await_or_execute(tmr.callback) [audio_listener-1] File "/opt/ros/humble/lib/python3.8/site-packages/rclpy/executors.py", line 107, in await_or_execute [audio_listener-1] return callback(*args) [audio_listener-1] File "/workspaces/isaac_ros-dev/build/audio_listener/audio_listener/audio_listener.py", line 55, in audio_publisher_timercallback [audiolistener-1] audio = self.stream.read(self.frames_perbuffer) [audio_listener-1] File "/home/admin/.local/lib/python3.8/site-packages/pyaudio/init.py", line 570, in read [audio_listener-1] return pa.read_stream(self._stream, num_frames, [audio_listener-1] OSError: [Errno -9981] Input overflowed [ERROR] [audio_listener-1]: process has died [pid 198921, exit code 1, cmd '/workspaces/isaac_ros-dev/install/audio_listener/lib/audio_listener/audio_listener --ros-args'].

managed to get the data from speaker on /audio_listener/audio but seems unreliable OSError: [Errno -9981] Input overflowed

mhubii commented 10 months ago

hi @robotx10 , any idea why this happens? Seems to be related to pyaudio

Have you tried only running the audio listener node? E.g.

ros2 run audio_listener audio_listener

then echo or hz to verify the audio works

ros2 topic hz /audio_listener/audio