Closed Hongbin98 closed 1 year ago
still waiting for your response :(
In the instruction_following.py, there are confused arguments like 'past_mimicit_path' and 'new_mimicit_path'.
When I try to set the arguments '--past' to None and '--new' to the dir of my DC files, it reports a size mismatch error.
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \ pipeline/train/instruction_following.py \ --pretrained_model_name_or_path='/home/linhb/hf_models/OTTER-MPT7B-Init' \ --new_mimicit_path="/home/linhb/dataset/MIMIC-IT/DC/DC_instructions.json" \ --new_images_path="/home/linhb/dataset/MIMIC-IT/DC/DC.json" \ --new_train_config_path="/home/linhb/dataset/MIMIC-IT/DC/DC_train.json" \ --batch_size=4 \ --num_epochs=9 \ --report_to_wandb \ --run_name=OTTER-MPT7B-densecaption \ --wandb_project=otter \ --workers=1 \ --lr_scheduler=cosine \ --learning_rate=1e-5 \ --warmup_steps_ratio=0.01
And the error is stack expects each tensor to be equal size, but got [1, 92, 3, 224, 224] at entry 0 and [1, 117, 3, 224, 224] at entry 1
still waiting for your response :(
In the instruction_following.py, there are confused arguments like 'past_mimicit_path' and 'new_mimicit_path'. When I try to set the arguments '--past' to None and '--new' to the dir of my DC files, it reports a size mismatch error.
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \ pipeline/train/instruction_following.py \ --pretrained_model_name_or_path='/home/linhb/hf_models/OTTER-MPT7B-Init' \ --new_mimicit_path="/home/linhb/dataset/MIMIC-IT/DC/DC_instructions.json" \ --new_images_path="/home/linhb/dataset/MIMIC-IT/DC/DC.json" \ --new_train_config_path="/home/linhb/dataset/MIMIC-IT/DC/DC_train.json" \ --batch_size=4 \ --num_epochs=9 \ --report_to_wandb \ --run_name=OTTER-MPT7B-densecaption \ --wandb_project=otter \ --workers=1 \ --lr_scheduler=cosine \ --learning_rate=1e-5 \ --warmup_steps_ratio=0.01
And the error is
stack expects each tensor to be equal size, but got [1, 92, 3, 224, 224] at entry 0 and [1, 117, 3, 224, 224] at entry 1
The new_mimicit_path
should be a naming issue. We will restore it to "mimicit_path" soon in a PR. If you do not have to put something back into current training, you can just ignore the past_mimicit_path
and relevant args.
still waiting for your response :(
In the instruction_following.py, there are confused arguments like 'past_mimicit_path' and 'new_mimicit_path'. When I try to set the arguments '--past' to None and '--new' to the dir of my DC files, it reports a size mismatch error.
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \ pipeline/train/instruction_following.py \ --pretrained_model_name_or_path='/home/linhb/hf_models/OTTER-MPT7B-Init' \ --new_mimicit_path="/home/linhb/dataset/MIMIC-IT/DC/DC_instructions.json" \ --new_images_path="/home/linhb/dataset/MIMIC-IT/DC/DC.json" \ --new_train_config_path="/home/linhb/dataset/MIMIC-IT/DC/DC_train.json" \ --batch_size=4 \ --num_epochs=9 \ --report_to_wandb \ --run_name=OTTER-MPT7B-densecaption \ --wandb_project=otter \ --workers=1 \ --lr_scheduler=cosine \ --learning_rate=1e-5 \ --warmup_steps_ratio=0.01
And the error is
stack expects each tensor to be equal size, but got [1, 92, 3, 224, 224] at entry 0 and [1, 117, 3, 224, 224] at entry 1
Expectedly we will resample the frames into a fixed range, so that would enable batchwise training.
You can modify the relevant function into
def process_dense_caption(self, instruction_id, instruction, answer, image_ids, in_context_example_ids, resample_frames=32):
patch_images = torch.tensor([])
all_texts = ""
all_instruction_ids = in_context_example_ids + [instruction_id]
random.shuffle(all_instruction_ids)
for cur_instruction_id in all_instruction_ids[:]:
cur_instruction = self.dataset[cur_instruction_id]["instruction"]
cur_instruction = self.pre_question(cur_instruction, self.max_src_length)
cur_answer = self.dataset[cur_instruction_id]["answer"]
cur_answer = self.pre_answer(cur_answer, self.max_tgt_length)
cur_text = f"User: {cur_instruction} GPT:<answer> {cur_answer}<|endofchunk|>"
all_texts += cur_text
all_texts = f"<image>{all_texts}"
# <image>User: {cur_incontext_instruction} GPT:<answer> {cur_incontext_answer}<|endofchunk|>User: {instruction} GPT:<answer> {answer}<|endofchunk|>
# <image>User: what does the image describe? GPT: XXX <|endofchunk|>User: Do you think this image is funny GPT:<answer> YYY <|endofchunk|>
image_ids = self.resample_frames(image_ids, resample_frames)
for cur_image_id in image_ids:
cur_image = self.images[cur_image_id]
cur_image = Image.open(BytesIO(base64.urlsafe_b64decode(cur_image))).convert("RGB")
cur_patch_image = self.patch_resize_transform(cur_image).unsqueeze(0)
if len(patch_images) == 0:
patch_images = cur_patch_image
else:
patch_images = torch.cat((patch_images, cur_patch_image))
patch_images = patch_images.unsqueeze(0)
return patch_images, all_texts
The resample_frames
is to uniformly extract fixed number of frames from different batch entries.
These all would be fixed in a next PR. You can update your current code manually or wait next PR and then pull them.
Thanks for your reply! Now I am succeed to run the training code of Otter.
If someone meet the same problem like me, you can also fix this issue by the following steps:
The training command in the README.MD is:
And I change the 'pretrained_model_name_or_path' and the path to mimicit:
However, it reports: 'mimicit_path', 'images_path' and 'train_config_path' are unrecognized arguments for instruction_following.py.
So, I wonder if there are any new training command for the latest training code.
Looking forward to your reply!