search

Luodian / Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
https://otter-ntu.github.io/
MIT License
3.55k stars 242 forks source link

About the CGD dataset #225

Closed zuwenqiang closed 1 year ago

zuwenqiang commented 1 year ago

Hi, great work and efforts!

I want to create my own CGD dataset based on my own data, but I have some confusion. To my knowledge, the final data we use to train is CGD_instruction.json. In the provided CGD_instruction.json, there are three values that need to be determined here, namely 'image_ids', 'instruction,answer', and 'rel_ins_ids'. Specifically, 'image_ids' refer to the two similar images selected for each turn conversation. 'instruction' refers to the question in each turn of the conversation, while 'answer' refers to the response in each turn. 'rel_ins_ids' refers to the related instructions in each turn of the conversation.

The question now is how to determine 'image_ids' and 'rel_ins_ids'. Although I have read all the documentation (including github and huggingface), it seems that there is no clear explanation on how to generate these data, and there are too many details left unclear. Besides, when I run 'python main.py --name change.SpotTheDifference', I need the 'annotations/spot_the_difference/train.json' file, but I am not sure what is this file and how to obtain it?

zuwenqiang commented 1 year ago

i just generate it successfully , just download coco ->train2017 is ok ,and fill the path, my cli is : python main.py --name=change.CocoGeneralDifference --image_path=/tf/data/LLaVA-Instruct-150K/train2017

I previously ran the "main.py" script in the "convert" folder using the argument "--name=change.CocoGeneralDifference", which resulted in the creation of a file named "CGD.json". My current issue is that I want to obtain a file named "CGD_instruction.json", so I need to run the "main.py" script in the "syphus" folder. However, I encountered a problem while running the script, which I mentioned in my previous description.

Luodian commented 1 year ago

hi we just add CGD prompts in main branch.

see the changes: https://github.com/Luodian/Otter/commit/824e00bcd3172c77449a4ab04f83bbbecaa8fab6

sorry for inconvenience, we forgot adding it previously.

The annotations should be in: https://entuedu-my.sharepoint.com/:f:/g/personal/libo0013_e_ntu_edu_sg/En3nLNmH1eJLvzGGvdGhhRUB6M_irbm4GngWB1BmwdmNMQ?e=v2WmrN

zuwenqiang commented 1 year ago

hi we just add CGD prompts in main branch.

see the changes: 824e00b

sorry for inconvenience, we forgot adding it previously.

The annotations should be in: https://entuedu-my.sharepoint.com/:f:/g/personal/libo0013_e_ntu_edu_sg/En3nLNmH1eJLvzGGvdGhhRUB6M_irbm4GngWB1BmwdmNMQ?e=v2WmrN

Thank you for your helpful response, the issue has been resolved now. However, is there a Python file or script here that generates "_query.json" such as csd_query.json?