Open wjfwjfwjf opened 1 year ago
it seems like I cannot use DC and SD to train together?
Traceback (most recent call last):
File "/mnt/disk2/home/wujianfeng/otter/Otter/pipeline/train/instruction_following.py", line 837, in
Yes, unless you set the batch_size to 1, you can not use image and video dataset together because they have different shapes.
Otter is designed to support multi-modal in-context instruction tuning based on the OpenFlamingo model, which involves conditioning the language model on the corresponding media, such as an image that corresponds to a caption or an instruction-response pair.
More specifically, Otter supports both image and video inputs, we interpret the input images in six dimensions as [B, N, T, C, W, H], where B represents the batch size, N and T correspond to the in-context images and video frames (with T=1 for images). The remaining dimensions, C,W,H, represent the image's channels, width, and height, respectively. By organizing the N and T dimensions, we can obtain in-context input samples tailored for both images and videos.
In your case, the T in SD and DC are different.
thank you, I want to know if I want to reproduce the result of image-text model, which dataset should I use?
thank you, I want to know if I want to reproduce the result of image-text model, which dataset should I use?
hi do you mean what result? For short answer is you could only use LA
subset to obtain good result, but it's not the best version of Otter.
We will have a revised Otter paper to demonstrate what datasets (more than 20 datasets we've tried) we used for instruction tuning and how do they help model gain knowledge and improve on different benchmarks.
So is there any update on this issue? what would be the best otter's data? Even for using solely the current MICMIC-IT data. I'm have been trying for so long and very confused about what data are IMAGE_TEXT, IMAGE_TEXT_IN_CONTEXT, and VIDEO_TEXT. Could there be a training data yaml just for the otter model be provided like the one in the Demo_Data.yaml. Thank you !
Before you open an issue, please check if a similar issue already exists or has been closed before.
When you open an issue, please be sure to include the following
dataset
(mimic-it download, usage, etc.),demo
(online demo),doc
(readme, wiki, paper, video etc.),evaluation
(evaluation result, performance of Otter etc.),model
(model configuration, components, etc.),train
(training configuration, process, code, etc.)Thank you for your contributions!