GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction. (arXiv:2311.14786v2 [cs.CV] UPDATED) https://ift.tt/GiSjLR4
Predicting pedestrian behavior is the key to ensure safety and reliability of
autonomous vehicles. While deep learning methods have been promising by
learning from annotated video frame sequences, they often fail to fully grasp
the dynamic interactions between pedestrians and traffic, crucial for accurate
predictions. These models also lack nuanced common sense reasoning. Moreover,
the manual annotation of datasets for these models is expensive and challenging
to adapt to new situations. The advent of Vision Language Models (VLMs)
introduces promising alternatives to these issues, thanks to their advanced
visual and causal reasoning skills. To our knowledge, this research is the
first to conduct both quantitative and qualitative evaluations of VLMs in the
context of pedestrian behavior prediction for autonomous driving. We evaluate
GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our
quantitative analysis focuses on GPT-4V's ability to predict pedestrian
behavior in current and future frames. The model achieves a 57% accuracy in a
zero-shot manner, which, while impressive, is still behind the state-of-the-art
domain-specific models (70%) in predicting pedestrian crossing actions.
Qualitatively, GPT-4V shows an impressive ability to process and interpret
complex traffic scenarios, differentiate between various pedestrian behaviors,
and detect and analyze groups. However, it faces challenges, such as difficulty
in detecting smaller pedestrians and assessing the relative motion between
pedestrians and the ego vehicle.
GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction. (arXiv:2311.14786v2 [cs.CV] UPDATED)
https://ift.tt/GiSjLR4
Predicting pedestrian behavior is the key to ensure safety and reliability of autonomous vehicles. While deep learning methods have been promising by learning from annotated video frame sequences, they often fail to fully grasp the dynamic interactions between pedestrians and traffic, crucial for accurate predictions. These models also lack nuanced common sense reasoning. Moreover, the manual annotation of datasets for these models is expensive and challenging to adapt to new situations. The advent of Vision Language Models (VLMs) introduces promising alternatives to these issues, thanks to their advanced visual and causal reasoning skills. To our knowledge, this research is the first to conduct both quantitative and qualitative evaluations of VLMs in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our quantitative analysis focuses on GPT-4V's ability to predict pedestrian behavior in current and future frames. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions. Qualitatively, GPT-4V shows an impressive ability to process and interpret complex traffic scenarios, differentiate between various pedestrian behaviors, and detect and analyze groups. However, it faces challenges, such as difficulty in detecting smaller pedestrians and assessing the relative motion between pedestrians and the ego vehicle.
via cs.RO updates on arXiv.org http://arxiv.org/