HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

This repository provides the official PyTorch implementation of the following paper:

[HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding]()
Zhaorun Chen^1,*, Zhuokai Zhao^1,*, Hongyin Luo ², Huaxiu Yao ³, Bo Li^1,4, Jiawei Zhou⁵

¹University of Chicago, ²Massachusetts Institute of Technology, ³UNC-Chapel Hill
⁴University of Illinois at Urbana-Champaign, ⁵Toyota Technological Institute at Chicago
_{* Equal contribution}

:partying_face: Features

:fire::fire: We are actively enlarging this repo to support more VLMs & decoders. Stay tuned! If you wish to add some algorithm/model or contribute to HALC, please feel free to reach out to us!

Currently supported online OH decoding methods

*: indicates the method supports beam search.

Currently supported post-hoc methods

:hammer_and_wrench: Installation

To install, run the following commands to install the required packages:

git clone https://github.com/BillChan226/HALC.git
cd HALC
conda env create -f environment.yml
conda activate halc

We employ Grounding DINO as the external detector to ground hallucinatory objects. To install GroundingDINO with CUDA, we simplify the installation process, where you can:

# set CUDA_HOME to the virtual environment halc
export CUDA_HOME=$CONDA_PREFIX
# install GroundingDINO
cd decoder_zoo/GroundingDINO
pip install -e .
# go back to HALC root
cd ../..

To download pre-trained model weights for DINO:

# default directory that contains the weights
mkdir model_checkpoints
cd model_checkpoints
# download weights
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
# go back to HALC root
cd ..

:bee: LVLM Backbones

The following evaluation requires for MSCOCO 2014 dataset. Please download here and extract it in your data path.

Besides, you need to prepare the following checkpoints of 7B base models:

• Download LLaVA-1.5 merged 7B model and specify it at Line 14 of eval_configs/llava-1.5_eval.yaml.
• Download LLaMA-2 7B model and specify it at Line 15 of minigpt4/configs/models/minigpt4_llama2.yaml.
• Download Vicuna 7B v1.1 model and specify it at Line 25 of minigpt4/configs/models/blip2_instruct_vicuna7b.yaml.
• Download Vicuna 7B v0 model and specify it at Line 18 of minigpt4/configs/models/minigpt4_vicuna0.yaml.
• Download MiniGPT-4 7B pretrained weights and specify it at Line 8 of eval_configs/minigpt4_eval.yaml.
• Download MiniGPT-4 7B pretrained weights for LlaMA-2 and specify it at Line 8 of eval_configs/minigpt4_llama2_eval.yaml.
• Download mPLUG-Owl2 7B pretrained weights and specify it at Line 14 of eval_configs/mplug-owl2_eval.yaml.

Arguments

Arguments for HALC

Arguments for OPERA

Arguments for VCD

:hourglass: Benchmarking OH

:chair: Running CHAIR evaluation for LVLMs object hallucination

Following Evaluating Object Hallucination in Large Vision-Language Models, we used "Please describe this image in detail." as the prompt to query LVLM for captions of the 500 images randomly sampled from COCO 2014 Val datast. Under root directory, run

python run_scripts/caption_generation.py --model [LVLM Backbone] --data_path [COCO_DIR] -d [Decoding Strategy] --num_samples 500 --seed [SEED] --gpu-id [GPU_IDs] --output_dir ./generated_captions/ --debugging 1

--debugging 1 will print the intermediate hallucination correction process of HALC.

:man_in_tuxedo: Running POPE evaluation for LVLMs object hallucination

Since OPOPE evaluates directly based on the caption generated for each image, it follows the caption generation procedure for CHAIR and differs in the subsequent metric calculation. To collect samples for the conventional POPE evaluation, under root directory, run

python run_scripts/pope_eval.py --model [LVLM Backbone] --data_path [COCO_DIR] -d [Decoding Strategy] --pope_type [random/popular/adversarial] --num_images 100 --seed [SEED] --gpu_id [GPU_IDs] --output_dir ./generated_captions/

:woman_juggling: Running MME Benchmark to evaluate LVLMs object hallucination

MME also follows the same procedure as CHAIR and OPOPE to collect samples. Alternatively, under root directory, run

python run_scripts/mme_eval.py --model [LVLM Backbone] --data_path [MME_DIR] -d [Decoding Strategy] --num_samples 30 --seed [SEED] --gpu-id [GPU_IDs] --output_dir ./generated_captions/

:hospital: Running post-hoc methods to revise generated captions

Under root directory, run

python run_scripts/reviser_eval.py -r [woodpecker/lure] --data_path [COCO_DIR] --c [PATH_TO_CAPTION] --seed [SEED] --gpu-id [GPU_IDs] --output_dir ./log/

Evaluation

CHAIR Scores

After preparing your caption files using the above commands, you can either choose to evaluate the captions in an one-shot mode (single caption) or batch mode (all the caption files in a folder). To evaluate a single caption file,

python eval/eval_hallucination.py --metric chair --chair_input_path [PATH_TO_CAPTION_DIR] -v

To evaluate a batch of caption files, run

python eval/caption_to_chair.py -c [PATH_TO_CAPTION_FOLDER_DIR]

to convert the caption files to the format ready for CHAIR evaluation in the same directory first. Then a _chair.json file will be produced under this folder. To further evaluate the CHAIR score as well as the generation quality scores, run

python eval/batch_eval.py -c [PATH_TO_CAPTION_FOLDER_DIR] --evaluator chair --coco_path [COCO_DIR]

Note that [COCO_DIR] is expected to contain both images and annotation files within the annotations subfolder. In other words, [COCO_DIR] should the the following structure:

COCO_DIR (val2014 for example)
  - annotations
    - captions_val2014.json
    - captions_val2014.json
    - instances_train2014.json
    - instances_val2014.json
    - person_keypoints_train2014.json
    - person_keypoints_val2014.json
  - COCO_val2014_000000000042.jpg
  - COCO_val2014_000000000073.jpg
  ...

POPE Scores

Similarly, you can also evaluate POPE in both modes. To evaluate a single caption file,

python eval_hallucination.py --metric pope --pope_answer_path [PATH_TO_CAPTION_DIR] --pope_question_path [PATH_TO_POPE_QUESTION] -v

To evaluate a batch of caption files, run

python eval/batch_eval.py -c [PATH_TO_CAPTION_FOLDER_DIR] --evaluator pope --pope_type [random/popular/adversarial]

The evaluation results will be saved in the same directory.

MME Scores

To evaluate the MME scores on each chosen subset, modify the subset_dir variable here to include the list of directories of your target directories and run

python eval/MME_eval.py

:wrench: Troubleshooting

Debugging HALC

Add --debugger argument to the command line to print the debugging output for HALC. Specifically, use --debugger 1 to print the intermediate hallucination correction with HALC, and --debugger 2 to print all the debugging output. For example:

python run_scripts/caption_generation.py --debugger 1

Error installing GroundingDINO

If error NameError: name '_C' is not defined is reported, refer to this issue for a quick fix.

Error installing pattern

conda install -c conda-forge pattern

CUDA Error installing GroundingDINO

conda install pytorch torchvision torchaudio pytorch-cuda=[YOUR NVIDIA CUDA VERSION] -c pytorch -c nvidia

runtimeError: Input type (float) and bias type (c10::Half) should be the same

simply reinstall torch==2.0.0 will most likely solve the issue

pip uninstall torch
pip install torch==2.0.0

:book: Acknowledgement

Please cite the paper as follows if you use the data or code from HALC:

@article{chen2024halc,
  title={HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding},
  author={Chen, Zhaorun and Zhao, Zhuokai and Luo, Hongyin and Yao, Huaxiu and Li, Bo and Zhou, Jiawei},
  journal={arXiv preprint arXiv:2403.00425},
  year={2024}
}

:book: Contact

Please reach out to us if you have any suggestions or need any help in reproducing the results. You can submit an issue or pull request, or send an email to zhaorun@uchicago.edu.

:key: License

This repository is under BSD 3-Clause License. Many codes are based on Lavis with BSD 3-Clause License here.