HPT - Open Multimodal Large Language Models

Hyper-Pretrained Transformers (HPT) is a novel multimodal LLM framework from HyperGAI, and has been trained for vision-language models that are capable of understanding both textual and visual inputs. HPT has achieved highly competitive results with state-of-the-art models on a variety of multimodal LLM benchmarks. This repository contains the open-source implementation of inference code to reproduce the evaluation results of HPT on different benchmarks.

Release

• [6/06] :fire: Releasing HPT 1.5 Edge, our latest open-source model tailored to edge devices. Despite its size (<5B), Edge demonstrates impressive capabilities while being extremely efficient. HPT 1.5 Edge is publicly available on [HuggingFace Repository]. Please read our [technical blog post] for more details.
• [5/03] HPT 1.5 Air, our best open-sourced 8B Multimodal LLM with Llama 3. Built with Meta Llama 3, Our hyper capable HPT 1.5 Air packs a punch on real world understanding and complex reasoning. HPT Air 1.5 achieves the best results among <10B models across a wide range of challenging benchmarks (MMMU, POPE, SEED-I, and more). HPT 1.5 Air is publicly available on [HuggingFace Repository]. Please read our [technical blog post] for more details.
• [3/16] HPT 1.0 Air is out, our most efficient model as a cost-effective solution that is capable of solving a wide range of vision-and-language tasks. HPT 1.0 Air is publicly available and achieves state-of-the-art results among all the open-source multimodal LLM models of similar or smaller sizes on the challenging MMMU benchmark. Please read our [technical blog post] and [HuggingFace Repository] for more details.

We release HPT 1.5 Edge as our latest open-sources model tailored to edge devices. Despite its size (<5B), Edge demonstrates impressive capabilities while being extremely efficient. We release HPT 1.5 Edge publicly at Huggingface and Github under the Apache 2.0 license.

Table of Contents

• Overview of Model Achitecture
• Quick Start
  • Installation
  • Prepare the Model
  • Demo
• Evaluations
• Benchmarks
• Pretrained Models Used
• Disclaimer and Responsible Use
• Contact Us
• License
• Acknowledgements

Overview of Model Achitecture

Quick Start

Installation

pip install -r requirements.txt
pip install -e .

Prepare the Model

You can download the model weights from HF into your [Local Path] and set the global_model_path as your [Local Path] in the model config file:

git lfs install
git clone https://huggingface.co/HyperGAI/HPT1_5-Edge [Local Path]

You can also set other strategies in the config file that are different from our default settings.

Demo

After setting up the config file, launch the model demo for a quick trial:

python demo/demo.py --image_path [Image]  --text [Text]  --model [Config]

Example:

python demo/demo.py --image_path demo/einstein.jpg  --text 'What is unusual about this image?'  --model hpt-edge-1-5

Evaluations

Launch the model for evaluation:

torchrun --nproc-per-node=8 run.py --data [Dataset] --model [Config]

Example for HPT 1.5 Edge:

torchrun --nproc-per-node=8 run.py --data MMMU_DEV_VAL --model hpt-edge-1-5

Benchmarks

For HPT 1.5 Edge

• The majority of the results presented are taken from the models‘ original reports while the others are from Phi-3-vision evaluations, which we mark with an asterisk (*).
• The benchmark result of HPT1.5 Air and HPT1.0 is in assets directory.

Pretrained Models Used

HPT 1.5 Edge

• Pretrained LLM: Phi-3-mini-4k-instruct

• Pretrained Visual Encoder: siglip-so400m-patch14-384

HPT 1.5 Air

• Pretrained LLM: Llama3-8B-Instruct

• Pretrained Visual Encoder: siglip-so400m-patch14-384

HPT 1.0 Air

• Pretrained LLM: Yi-6B-Chat

• Pretrained Visual Encoder: clip-vit-large-patch14-336

Disclaimer and Responsible Use

Note that the HPT Air is a quick open release of our models to facilitate the open, responsible AI research and community development. It does not have any moderation mechanism and provides no guarantees on their results. We hope to engage with the community to make the model finely respect guardrails to allow practical adoptions in real-world applications requiring moderated outputs.

Contact Us

• Contact: HPT@hypergai.com
• Follow us on Twitter.
• Follow us on Linkedin.
• Visit our website to learn more about us.

License

This project is released under the Apache 2.0 license. Parts of this project contain code and models from other sources, which are subject to their respective licenses and you need to apply their respective license if you want to use for commercial purposes.

Acknowledgements

The evaluation code for running this demo was extended based on the VLMEvalKit project. We also thank OpenAI for open-sourcing their visual encoder models, 01.AI, Meta and Microsoft for open-sourcing their large language models.