LLM Testing 1 - Read up on GPT-4o for Vision Benchmark

[duncancalvert]

1. Read up on GPT-4o vision accuracy, precision, and recall benchmarks for ceiling baselines. Document research papers on our paper.

[duncancalvert]

Recommended vision language models/approaches: -Benchmarks:

Vision understanding evals - GPT-4o achieves state-of-the-art performance on visual perception benchmarks. All vision evals are 0-shot, with MMMU, MathVista, and ChartQA as 0-shot CoT. Reference: https://openai.com/index/hello-gpt-4o/

Closed Architecture:

• Gemini 1.5 Flash $0.35 for 1M input - tested this and it can't process images.
• GPT-4o - tested with 4 uploaded photos and it scored 100% with determining the object I was holding.

Open Source

• fine-tuning LLaVA with LoRA for enhanced spatial reasoning
• prismatic-vlms through some robotics-related works: openvla, embodiedCoT
• Llava model within Ollama

Papers to review:

• Real-time Vision-based Navigation for a Robot in an Indoor Environment: https://arxiv.org/abs/2307.00666