search

jacklishufan / InstructAny2Pix

25 stars 0 forks source link

IndexError: index 0 is out of bounds for dimension 0 with size 0 #9

Open KOjuny opened 2 months ago

KOjuny commented 2 months ago

Hello. I am interested in your project and have been conducting various experiments with it. However, I encountered the following error and do not know how to resolve it. This error occurs in pipeline.py at line 228. I would appreciate your help.

IndexError: index 0 is out of bounds for dimension 0 with size 0

jacklishufan commented 2 months ago

Hi. This suggests the model did not output any image tokens. I suggest the first step is to go over our demo notebook and try examples in it https://github.com/jacklishufan/InstructAny2Pix/blob/main/Demo-Retrained.ipynb

If this works, the issue is with the input format. Can you provide more details on what kind of instruction you are trying? If you are expecting pure language output (e.g. music captioning task), please pass in llm_only=True in the forward function.

KOjuny commented 2 months ago

Thanks for replying. My instruction was add <video> to <video>. And input image was bear statue (NeRF data), input sound was footstep on snow. This error sometime appear but almost time do not appear.

KOjuny commented 2 months ago

I have one more error. I input all the same inputs but instruction change <video> to <video> and change bear in <video> to <video>. Instruct to change is not available?? 


  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/demo.py", line 61, in <module>
    main(args.image_dir, args.audio_dir, args.audio_num, args.output_folder, args.alpha, args.norm, args.refinement, args.num_inference_steps, args.cfg, args.inst, args.ans)
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/demo.py", line 27, in main
    res0, res, output_caption = pipe(
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/ia2n/instructany2pix/pipeline.py", line 310, in __call__
    image_embeds,base_embed,output_caption,base_img_path,extra_data = self.forward_llm(inst,mm_data,use_cache=use_cache)
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/ia2n/instructany2pix/pipeline.py", line 255, in forward_llm
    gen_idx = out_seq.view(-1).tolist().index(base_tkn)+1
ValueError: 32007 is not in list```
jacklishufan commented 2 months ago

I suggest try prompts: Modify video by replacing the bear with video, Replace the bear in video with video, Remove the bear in video and add video instead, etc This seems to be a small bias in our training data, but we did not use the term "change" a lot.

KOjuny commented 2 months ago

Thanks for replying. I'm really sorry, but the following error occurred again. I input the same instruction and the same sound. I only used a slightly different image. How can i solve this problem??

Traceback (most recent call last):
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/bin/ns-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/engine/trainer.py", line 261, in train
    loss, loss_dict, metrics_dict = self.train_iteration(step)
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/node_data/urp24s_jsko/anaconda3/envs/ia2n/lib/python3.10/site-packages/nerfstudio/engine/trainer.py", line 496, in train_iteration
    _, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/ia2n/ia2n_pipeline.py", line 225, in get_train_loss_dict
    ],
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/ia2n/instructany2pix/pipeline.py", line 310, in __call__
    image_embeds,base_embed,output_caption,base_img_path,extra_data = self.forward_llm(inst,mm_data,use_cache=use_cache)
  File "/node_data/urp24s_jsko/sound_in2n/instructany2nerf/ia2n/instructany2pix/pipeline.py", line 228, in forward_llm
    gen_idx = all_gen_tokens[0]
IndexError: index 0 is out of bounds for dimension 0 with size 0
jacklishufan commented 2 months ago

Hi, can you clarify what is the image and instructions? (If you could share the image, that would be the best). Also, can you clarify what "same instruction and the same sound" refers to? Is it the same instruction and the same sound as our demo? or the same as another working example of yours?

Further details of the instruction can help us diagnose the problem