Where's the data at for the shared script for OtterHD?

[StrangeTcy]

So far the only way to run inference on OtterHD I've seen is with a script from shared_scripts/Demo_OtterHD.sh. This script accepts multiple args, including --training_data_yaml=./Demo_Data.yaml, and Demo_Data.yaml contains lines like

mimicit_path: azure_storage/json/LA/LADD_instructions.json

Now I don't have any azure storage, so when I try running that script it fails. My suggestions are:

1. Provide explicit instructions for getting the datasets necessary and editing the Demo_Data.yaml accordingly (leaving aside for now why you'd need all the datasets for a something which is a Demo anyway)
2. Provide an official way to run demo inference with OtterHD (at the moment of writing this issue the only instruction published seems to be about finetuning (https://github.com/Luodian/Otter/blob/main/docs/OtterHD.md#how-to-finetune), and I think something like https://github.com/Luodian/Otter/blob/main/pipeline/serve/gradio_web_server.py can be adapted nicely)

[Luodian]

Hi Thanks for this suggestion. I think you could visit the files at our onedrive space: https://entuedu-my.sharepoint.com/personal/libo0013_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Flibo0013%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2FMIMICIT%5FParquets&ga=1

And also you could check the markdown file here to see how to generate data files.

https://github.com/Luodian/Otter/blob/main/docs/mimicit_format.md

We would also provide better inference scripts later, for now you could check here to see how we benchmark OtterHD. https://github.com/Luodian/Otter/blob/main/pipeline/benchmarks/models/otterhd.py

[StrangeTcy]

Hi Thanks for this suggestion. I think you could visit the files at our onedrive space: https://entuedu-my.sharepoint.com/personal/libo0013_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Flibo0013%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2FMIMICIT%5FParquets&ga=1

-- thanks, but no I can't; I'd have to login with a microsoft account first (done) and then visit a link to a shared file or folder. At the moment the error is

We're sorry, but [my email here] can't be found in the entuedu-my.sharepoint.com directory. Please try again later, while we try to automatically fix this for you. 

And also you could check the markdown file here to see how to generate data files.

https://github.com/Luodian/Otter/blob/main/docs/mimicit_format.md

Ok, this readme contains working links to OneDrive; I'd just prefer some explicit instructions that'd say something along the lines of "Go to OneDrive. download all the files, edit Demo_Data.yaml accordingly or use this neat script to do all this automatically". The question of why I'd need all the training/finetuning data to run inference still stands, though

We would also provide better inference scripts later, for now you could check here to see how we benchmark OtterHD. https://github.com/Luodian/Otter/blob/main/pipeline/benchmarks/models/otterhd.py

Well, that's great, but it's just a definition of the OtterHD class. The actual evaluation script is https://github.com/Luodian/Otter/blob/main/pipeline/benchmarks/evaluate.py, and even it doesn't look as useful as, say, https://github.com/Luodian/Otter/blob/main/pipeline/serve/cli.py In fact, the latter might just work with OtterHD, it just won't be as nice and visual as a gradio service

[Luodian]

The benchmark eval script is here: https://github.com/Luodian/Otter/blob/main/docs/benchmark_eval.md

For those in Demo_Data.yaml, it's mainly used for training, not inference.

And sorry for misunderstanding caused by these files in serve. We are not using them anymore, it's previously used for host Otter/Otter-Video model.

We are now using a front/backend separated way to host OtterHD model in an endpoint way (need to configure Cloudflared service).

https://github.com/Luodian/Otter/blob/main/pipeline/serve/deploy/otterhd_endpoint.py

I am sorry it's a little bit hard to go through them for new comers, I will make a new PR to refactor them and provide better docs maybe within next week.

[Luodian]

With above benchmark_eval.md, I think you could have the sense how to run Fuyu/OtterHD (as well as other models) for benchmarking.

And with the generate function, you could modify it yourself for inference on other dataset or just image-query pairs.

[Luodian]

As for the access to the data files in Onedrive, I think I've already grant all open access for anyone.

[StrangeTcy]

A single clear script to download all the files mentioned in Demo_Data.yaml would be appreciated. here's the file: https://github.com/Luodian/Otter/blob/f6606c9cf1a3a955fcceaf535ae31afc935278ff/shared_scripts/Demo_Data.yaml

You can get the LADD_instructions.json, say, from huggingface -- https://huggingface.co/datasets/pufanyi/MIMICIT/tree/main/data/LA The M3IT files aren't present anywhere, and the files in the original M3IT are different: https://huggingface.co/datasets/MMInstruction/M3IT/tree/main/data/captioning/coco instructions.json vs coco_instructions.json The Parquet files are nowhere to be found -- correction, there's https://entuedu-my.sharepoint.com/personal/libo0013_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Flibo0013%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2FMIMICIT%5FParquets&ga=1, and it has no files from the M3IT part of Demo_Data.yaml

[StrangeTcy]

Ok, we now can get most of the files we need from huggingface, I'm just not sure which thingy_instructions.json files should match which thingy.parquet files.