The output of the largest model seems unreliable.

[MufanColin]

We are running the XXL Unified IO 2 model (pytorch version) on a NVIDIA GeForce RTX 4090. The demo.py file is shown below:

from uio2 import config
from uio2.model import UnifiedIOModel
from uio2.preprocessing import UnifiedIOPreprocessor
preprocessor = UnifiedIOPreprocessor.from_config(config.XXL, "tokenizer.model")
model = UnifiedIOModel(config.XXL)

from uio2.runner import TaskRunner
runner = TaskRunner(model, preprocessor)

for _ in range(3):
    ans1 = runner.categorization("something.jpg", ["boy", "girl", "cat", "dog", "car", "plane", "sun"])
    print(f"Prediction: {ans1}")

something.jpg file is actually a picture of cat.

The output is shown below:

Prediction: sun
Prediction: sun
Prediction: sun

Are we doing anything wrong?

[shubhamagarwal92]

I guess config.XXL model initialized the model from scratch. Maybe try

preprocessor = UnifiedIOPreprocessor.from_pretrained("allenai/uio2-preprocessor", tokenizer="/path/to/tokenizer")
model = UnifiedIOModel.from_pretrained("allenai/uio2-large")