search

OpenRobotLab / PointLLM

[ECCV 2024 Best Paper Candidate] PointLLM: Empowering Large Language Models to Understand Point Clouds
https://runsenxu.com/projects/PointLLM
555 stars 24 forks source link
3d chatbot foundation-models gpt-4 large-language-models llama multimodal objaverse point-cloud pointllm representation-learning vision-and-language

readme


PointLLM: Empowering Large Language Models to Understand Point Clouds

Runsen Xu  Xiaolong Wang  Tai Wang  Yilun Chen  Jiangmiao Pang*  Dahua Lin 
The Chinese University of Hong Kong Shanghai AI Laboratory Zhejiang University

🏠 About

Dialogue_Teaser

We introduce PointLLM, a multi-modal large language model capable of understanding colored point clouds of objects. It perceives object types, geometric structures, and appearance without concerns for ambiguous depth, occlusion, or viewpoint dependency. We collect a novel dataset comprising 660K simple and 70K complex point-text instruction pairs to enable a two-stage training strategy. To rigorously evaluate our model's perceptual abilities and its generalization capabilities, we establish two benchmarks: Generative 3D Object Classification and 3D Object Captioning, assessed through three different evaluation methods.

🔥 News

📋 Contents

🤖 Online Demo

PointLLM is online! Try it at http://101.230.144.196 or at OpenXLab/PointLLM.

You can chat with PointLLM about the models of the Objaverse dataset or about your own point clouds!

Please do not hesitate to tell us if you have any feedback! 😃

💬 Dialogue Examples

Dialogue 1 Dialogue 2 Dialogue 3 Dialogue 4

🔍 Overview

Model

The point encoder extracts features from the input point cloud and projects them to the latent space of the LLM backbone. The LLM backbone processes sequences of point tokens and text tokens, and generates the predicted tokens as the output.

Experiment Results

Quantitative Comparisons with baselines.

Please refer to our paper for more results.

!!!Note: Traditional metrics such as BLEU-1, ROUGE-L, and METEOR tend to favor shorter responses and may not effectively capture semantic accuracy. For a detailed discussion on this, please refer to our paper. We suggest the community not solely rely on these metrics for evaluation.

Qualitative Comparisons with baselines.

Please refer to our paper for more results.

📦 Training and Evaluation

Installation

We test our codes under the following environment:

To start:

  1. Clone this repository. 
    git clone git@github.com:OpenRobotLab/PointLLM.git
cd PointLLM
  2. Install packages 
    
conda create -n pointllm python=3.10 -y
conda activate pointllm
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

* for training

pip install ninja pip install flash-attn


### Data Preparation
#### Objaverse Training Data
1. Download the two compressed files of 660K Objaverse colored point clouds [here](https://huggingface.co/datasets/RunsenXu/PointLLM/tree/main). They require about 77GB of storage space.
2. Run the following command to merge the two files into one and uncompress it. This will produce a folder named `8192_npy` containing 660K point cloud files named `{Objaverse_ID}_8192.npy`. Each file is a numpy array with dimensions (8192, 6), where the first three dimensions are `xyz` and the last three dimensions are `rgb` in [0, 1] range.
```bash
cat Objaverse_660K_8192_npy_split_a* > Objaverse_660K_8192_npy.tar.gz
tar -xvf Objaverse_660K_8192_npy.tar.gz
  1. In PointLLM folder, create a folder data and create a soft link to the uncompressed file in the directory. 
    cd PointLLM
mkdir data
ln -s /path/to/8192_npy data/objaverse_data

Instruction-Following Data

  1. In PointLLM/data folder, create a directory named anno_data.
  2. Our instruction-following data, including both the simple-description and complex instructions, can be downloaded here. If you have difficulty downloading the data (e.g. network issue), please email the authors.
    • The simple-description data has 660K samples and the complex instructions have 70K samples.
    • Both training data are based on the Objaverse dataset.
    • The complex instructions are generated with GPT-4.
  3. Put the data files in the anno_data directory. The directory should look like this: 
    PointLLM/data/anno_data
├── PointLLM_brief_description_660K_filtered.json
├── PointLLM_brief_description_660K.json
└── PointLLM_complex_instruction_70K.json
  4. Note, the PointLLM_brief_description_660K_filtered.json is filtered from PointLLM_brief_description_660K.json by removing the 3000 objects we reserved as the validation set. If you want to reproduce the results in our paper, you should use the PointLLM_brief_description_660K_filtered.json for training. The PointLLM_complex_instruction_70K.json contains objects from the training set.
  5. If you want to generate the complex instructions by yourself, please refer to our paper for other details. The system prompt is at pointllm/data/data_generation/system_prompt_gpt4_0613.txt.

Evaluation Data

  1. Download the referencing GT PointLLM_brief_description_val_200_GT.json we use for the benchmarks on Objaverse dataset here, and put it in PointLLM/data/anno_data. We also provide the 3000 object ids we filter during training here and their corresponding referencing GT here, which can be used to evaluate on all the 3000 objects.
  2. Create a directory named modelnet40_data in PointLLM/data. Download the test split of ModelNet40 point clouds modelnet40_test_8192pts_fps.dat here and put it in PointLLM/data/modelnet40_data.

Training

Download the Initial LLM and Point Encoder Weights

  1. In PointLLM folder, create a directory named checkpoints.
  2. Download the pre-trained LLM and point encoder: PointLLM_7B_v1.1_init or PointLLM_13B_v1.1_init. Put them in the checkpoints directory.
  3. Note that the above "v1.1" means we use the Vicuna-v1.1 checkpoints, and you do not need to download the original LLaMA weights again.

Start Training

  1. For stage-1 training, simply run: 
    cd PointLLM
scripts/PointLLM_train_stage1.sh
  2. After stage-1 training, start stage-2 training: 
    scripts/PointLLM_train_stage2.sh

PointLLM-v1.1 and PointLLM-v1.2

Usually, you do not have to care about the following contents. They are only for reproducing the results in our v1 paper (PointLLM-v1.1). If you want to compare with our models or use our models for downstream tasks, please use PointLLM-v1.2 (refer to our v2 paper), which has better performance.

The following steps are for reproducing PointLLM-v1.1 (click to expand) 1. PointLLM v1.1 and v1.2 use slightly different pre-trained point encoders and projectors. If you want to reproduce PointLLM v1.1, edit the `config.json` file in the directory of initial LLM and point encoder weights, for example, `vim checkpoints/PointLLM_7B_v1.1_init/config.json`. 2. Change the key `"point_backbone_config_name"` to specify another point encoder config: ```bash # change from "point_backbone_config_name": "PointTransformer_8192point_2layer" # v1.2 # to "point_backbone_config_name": "PointTransformer_base_8192point", # v1.1 ``` 3. Edit the checkpoint path of the point encoder in `scripts/train_stage1.sh`: ```bash # change from point_backbone_ckpt=$model_name_or_path/point_bert_v1.2.pt # v1.2 # to point_backbone_ckpt=$model_name_or_path/point_bert_v1.1.pt # v1.1 ```

Chatting

  1. The trained model checkpoints are available here (including different versions of PointLLM).
  2. Run the following command to launch a chatbot using the torch.float32 data type for chatting about 3D models of Objaverse. The model checkpoints will be downloaded automatically. You can also manually download the model checkpoints and specify their paths. Here is an example: 
    cd PointLLM
PYTHONPATH=$PWD python pointllm/eval/PointLLM_chat.py --model_name RunsenXu/PointLLM_7B_v1.2 --data_name data/objaverse_data --torch_dtype float32
  3. You can also easily modify the codes for using point clouds other than those from Objaverse, as long as the point clouds input to the model have dimensions (N, 6), where the first three dimensions are xyz and the last three dimensions are rgb (in [0, 1] range). You may sample the point clouds to have 8192 points, as our model is trained on such point clouds.

  4. The following table shows GPU requirements for different models and data types. We recommend using torch.bfloat16 if applicable, which is used in the experiments in our paper.

    Model Data Type GPU Memory
    PointLLM-7B torch.float16 14GB
    PointLLM-7B torch.float32 28GB
    PointLLM-13B torch.float16 26GB
    PointLLM-13B torch.float32 52GB

Gradio Demo

  1. We provide the codes for our online Gradio demo. You can run the following commands to launch the demo locally for chatting and visualization. 
    cd PointLLM
PYTHONPATH=$PWD python pointllm/eval/chat_gradio.py --model_name RunsenXu/PointLLM_7B_v1.2 --data_name data/objaverse_data
  2. Kind remind: if you want to release the demo in public, please refer to https://www.gradio.app/guides/sharing-your-app#security-and-file-access.

Evaluation

Inferencing

  1. Run the following commands to infer the results.
  2. Different commands for inferencing on different benchmarks (PointLLM_7B_v1.2 as an example): 
    
cd PointLLM
export PYTHONPATH=$PWD

Open Vocabulary Classification on Objaverse

python pointllm/eval/eval_objaverse.py --model_name RunsenXu/PointLLM_7B_v1.2 --task_type classification --prompt_index 0 # or --prompt_index 1

Object captioning on Objaverse

python pointllm/eval/eval_objaverse.py --model_name RunsenXu/PointLLM_7B_v1.2 --task_type captioning --prompt_index 2

Close-set Zero-shot Classification on ModelNet40

python pointllm/eval/eval_modelnet_cls.py --model_name RunsenXu/PointLLM_7B_v1.2 --prompt_index 0 # or --prompt_index 1

3. Please check the default command-line arguments of these two scripts. You can specify different prompts, data paths, and other parameters. 
4. After inferencing, the results will be saved in `{model_name}/evaluation` as a dict with the following format:
```bash
{
  "prompt": "",
  "results": [
    {
      "object_id": "",
      "ground_truth": "", 
      "model_output": "",
      "label_name": "" # only for classification on modelnet40
    }
  ]
}

ChatGPT/GPT-4 Evaluation

  1. Get your OpenAI API key at https://platform.openai.com/api-keys.
  2. Run the following commands to evaluate the model outputs in parallel with ChatGPT/GPT-4 (which cost approximately $1.5 to $2.2 USD). 
    
cd PointLLM
export PYTHONPATH=$PWD
export OPENAI_API_KEY=sk-****

Open Vocabulary Classification on Objaverse

python pointllm/eval/evaluator.py --results_path /path/to/model_output --model_type gpt-4-0613 --eval_type open-free-form-classification --parallel --num_workers 15

Object captioning on Objaverse

python pointllm/eval/evaluator.py --results_path /path/to/model_output --model_type gpt-4-0613 --eval_type object-captioning --parallel --num_workers 15

Close-set Zero-shot Classification on ModelNet40

python pointllm/eval/evaluator.py --results_path /path/to/model_output --model_type gpt-3.5-turbo-0613 --eval_type modelnet-close-set-classification --parallel --num_workers 15

3. The evaluation script supports interruption and resumption. You can interrupt the evaluation process at any time by using `Ctrl+C`. This will save the temporary results. If an error occurs during the evaluation, the script will also save the current state. You can resume the evaluation from where it left off by running the same command again.
4. The evaluation results will be saved in `{model_name}/evaluation` as another dict.
Some of the metrics are explained as follows:
```bash
"average_score": The GPT-evaluated captioning score we report in our paper.
"accuracy": The classification accuracy we report in our paper, including random choices made by ChatGPT when model outputs are vague or ambiguous and ChatGPT outputs "INVALID".
"clean_accuracy": The classification accuracy after removing those "INVALID" outputs.
"total_predictions": The number of predictions.
"correct_predictions": The number of correct predictions.
"invalid_responses": The number of "INVALID" outputs by ChatGPT.

# Some other statistics for calling OpenAI API
"prompt_tokens": The total number of tokens of the prompts for ChatGPT/GPT-4.
"completion_tokens": The total number of tokens of the completion results from ChatGPT/GPT-4.
"GPT_cost": The API cost of the whole evaluation process, in US Dollars 💵.
  1. Open-Step Evaluation. You can also start evaluation immediately after inferencing by passing the --start_eval flag and specifying the --gpt_type. For example: 
    python pointllm/eval/eval_objaverse.py --model_name RunsenXu/PointLLM_7B_v1.2 --task_type classification --prompt_index 0 --start_eval --gpt_type gpt-4-0613

Traditional Metric Evaluation

  1. For the object captioning task, run the following command to evaluate model outputs with traditional metrics including BLEU, ROUGE, METEOR, Sentence-BERT, and SimCSE. 
    python pointllm/eval/traditional_evaluator.py --results_path /path/to/model_captioning_output
  2. Note that we recommend not using BLEU, ROUGE, and METEOR for evaluation as they favor short captions and fall short of capturing semantic accuracy and diversity.

📝 TODO List

Community contributions are welcome!👇 If you need any support, please feel free to open an issue or contact us.

🔗 Citation

If you find our work and this codebase helpful, please consider starring this repo 🌟 and cite:

@article{xu2023pointllm,
  title={PointLLM: Empowering Large Language Models to Understand Point Clouds},
  author={Xu, Runsen and Wang, Xiaolong and Wang, Tai and Chen, Yilun and Pang, Jiangmiao and Lin, Dahua},
  journal={arXiv preprint arXiv:2308.16911},
  year={2023}
}

📄 License

Creative Commons License
This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

📚 Related Work

Together, Let's make LLM for 3D great!

👏 Acknowledgements