The dataset and code of the KDD23 paper 'A Study of Situational Reasoning for Traffic Understanding' See the full paper here
This repository contains four datasets: tv.jsonl
, bdd.jsonl
, hdt.jsonl
and manuals.jsonl
. All datasets are structured in a JSON Lines format, where each line is a separate JSON object corresponding to a single data point or example.
The Complex-TV-QA dataset, to our knowledge, is the inaugural resource that provides human-annotated, detailed video captions within traffic scenarios, alongside complex reasoning questions. This novel dataset not only stands as a vital tool for evaluating language models in real-world video-QA and video-reasoning research, but also offers valuable insights for the development and understanding of multi-modal video reasoning models and related works.
The tv.jsonl
dataset has the following fields:
BDD-QA is distinguished by its encompassing range of traffic actions, crafted to rigorously evaluate a model's decision-making abilities in traffcu senario. This makes it a potent tool for high-level decision-making research within traffic contexts, including autonomous driving developments.
The bdd.jsonl
dataset has the following fields:
HDT-QA, coupled with driving manuals, offers an extensive compendium of driving instructions and driving knowledge tests across all 51 states of the US. This resource is beneficial for assessing the incorporation and impact of traffic knowledge within intelligent driving systems, marking a crucial stride towards more advanced, informed, and safe autonomous driving technology.
The hdt.jsonl
dataset has the following fields:
The manuals.jsonl
dataset is a large collection of driving manuals from 51 states of the US, which is oringally crawled from DMV TEST PRO, which contains the following keys:
CDL
, permit
, motorcycle
In the code
folder we have the evaluation code for the experiments in the paper.
Firstly, clone this repo to you local folder:
git clone https://github.com/saccharomycetes/text-based-traffic-understanding.git
cd text-based-traffic-understanding
Then you can install the required packages by
pip install -r requirements.txt
Please ensure these are installed before running the script.
The script can be run using the following command:
python qa_retrieve_eval.py
--data_dir [DATA_DIR]
--output_dir [OUTPUT_DIR]
--num_related [NUM_RELATED]
--corpus_file [CORPUS_FILE]
--model_dir [MODEL_DIR]
Replace the bracketed terms with the appropriate paths or values:
[DATA_DIR]
: Directory of the data file that you want to evaluate[OUTPUT_DIR]
: Directory where the results will be saved[NUM_RELATED]
: number of retrieved documents you want to retrieve, set it to 0 if you want to run the QA-only evaluation[CORPUS_FILE]
: Path to the corpus file,which is our manuals.jsonl
dataset[MODEL_DIR]
: Directory of the pre-trained model, in our evaluation, we use the Unified-QA-V2The script can be run using the following command:
python kg_eval.py
--data_dir [DATA_DIR]
--output_dir [OUTPUT_DIR]
--model_dir [MODEL_DIR]
--eval_batch_size [EVAL_BATCH_SIZE]
Replace the bracketed terms with the appropriate paths or values:
[DATA_DIR]
: Directory of the data file that you want to evaluate[OUTPUT_DIR]
: Directory where the results will be saved[MODEL_DIR]
: Directory of the pre-trained model (we haven't uploaded this model for now, will be released soon)[EVAL_BATCH_SIZE]
: Batch size for evaluationThe script can be run using the following command:
python nli_eval.py
--data_dir [DATA_DIR]
--output_dir [OUTPUT_DIR]
--model_dir [MODEL_DIR]
--eval_batch_size [EVAL_BATCH_SIZE]
Replace the bracketed terms with the appropriate paths or values:
[DATA_DIR]
: Directory of the data file that you want to evaluate[OUTPUT_DIR]
: Directory where the results will be saved[MODEL_DIR]
: Directory of the pre-trained model, for our NLI evaluation, we use the Roberta-large-mnli[EVAL_BATCH_SIZE]
: Batch size for evaluationIf you find our dataset or code to be useful in your research, please consider citing the following paper:
@article{zhang2023study,
title={A Study of Situational Reasoning for Traffic Understanding},
author={Zhang, Jiarui and Ilievski, Filip and Ma, Kaixin and Kollaa, Aravinda and Francis, Jonathan and Oltramari, Alessandro},
journal={arXiv preprint arXiv:2306.02520},
year={2023}
}
jrzhang [AT] isi.edu