We propose LENS, a modular approach for tackling computer vision problems byleveraging the power of large language models (LLMs). Our system uses alanguage model to reason over outputs from a set of independent and highlydescriptive vision modules that provide exhaustive information about an image.We evaluate the approach on pure computer vision settings such as zero- andfew-shot object recognition, as well as on vision and language problems. LENScan be applied to any off-the-shelf LLM and we find that the LLMs with LENSperform highly competitively with much bigger and much more sophisticatedsystems, without any multimodal training whatsoever. We open-source our code athttps://github.com/ContextualAI/lens and provide an interactive demo.
URL
Affiliations
Abstract
Translation (by gpt-3.5-turbo)
Summary (by gpt-3.5-turbo)