search

vikhyat / moondream

tiny vision language model
https://moondream.ai
Apache License 2.0
5.14k stars 447 forks source link

Support running eval scripts on multiple GPUs #69

Open vikhyat opened 7 months ago

vikhyat commented 7 months ago

I'm starting to implement benchmarking scripts directly in this repository for reproducibility. We should support running them on multiple GPUs when available since they take a fair bit of time to run right now.

e.g. https://github.com/vikhyat/moondream/blob/main/moondream/eval/tallyqa.py

kyleboddy commented 6 months ago

Ran the commands to download the dataset and had to move data/ into the root folder along with tallyqa.py and change the import to import moondream from moondream as I was running into relative import errors.

Ripping this overnight on an RTX 4090 and a OC'd Ryzen 7 5800X. Will post results if it runs successfully.

kyleboddy commented 6 months ago

Welp this doesn't seem right, but I ran it.

root@bcf0ce95fd8a:~/moondream# python3 tallyqa.py
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3.72G/3.72G [00:33<00:00, 110MB/s]
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 69.0/69.0 [00:00<00:00, 625kB/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2412/2412 [25:02<00:00,  1.61it/s]
Simple: 22991, Correct: 0, Accuracy: 0.0
Total: 38589, Correct: 0, Accuracy: 0.0
muhd360 commented 1 month ago

hi i would like to be assigned this issue @vikhyat