Closed lalulxm closed 3 months ago
Thanks for your interest!
When early truncation appears, it is recommended to reduce penalty_weights
like penalty_weights=0.1
, or reduce length_penalty
in generate args. These will help to avoid early truncation in beam search.
Thank you for your early reply!
I have tried to modify the penalty_weights
and length_penalty
parameters, and the part of the code that calls the generate function is shown below:
out = model.generate(
{"image": norm(image), "prompt": qu},
use_nucleus_sampling=args.sample,
num_beams=5,
max_new_tokens=4096,
output_attentions=True,
length_penalty=0.1,
opera_decoding=True,
scale_factor=50,
threshold=15,
num_attn_candidates=5,
penalty_weights=0.1,
)
but the re-inference still results in 1/5 truncations.
Maybe we should also reduce the scale_factor, like:
out = model.generate(
{"image": norm(image), "prompt": qu},
use_nucleus_sampling=args.sample,
num_beams=5,
max_new_tokens=4096,
output_attentions=True,
length_penalty=0.1,
opera_decoding=True,
scale_factor=20,
threshold=15,
num_attn_candidates=5,
penalty_weights=0.1,
)
Hope it works well :)
Thank you for your great work! I tried to inference with my own model based on llava1.5 fine-tuning plus OPERA's method, but about 1/3 of the results of the inferencing appear to be truncated early. I followed the method in demo.ipynb and changed Line 14 of
eval_configs/llava-1.5_eval.yaml
to the path of the fine-tuned model. With beam=5 and max_new_tokens=1024, but the inference results showed the following early truncation:How can I solve this problem?