Unable to reproduce benchmark results on LLaVA-v1.5-7b

[ZQ1102118381]

Hi,

Thank you for your nice work.

I have a trouble in replicating the benchmark result like this:

• "paper-Tokenpacker" is the papers result
• "open-checkpoint" uses the checkpoint of TokenPacker-7b
• “ours" means reproduce result.

We trained with 8 A100, However, the evaluation results on other datasets seem reasonable.

I used the same details as in the provided code.

[typ1012]

We encountered the same problem.

[LiWentomng]

@ZQ1102118381 @typ1012 Thanks for pointing out this issue. The current code combines the patch slicing and tokenpacker, which may lead to some errors for the only original tokenpacker. We will carefully review the code.

[LiWentomng]

@ZQ1102118381 @typ1012 We carefully review our codes, and everything appears to be well. We re-train models with the same codes, we noticed some fluctuations in performance. We listed the model performance we have re-trained for reference.

Additionally, we have provided a branch that only contains TokenPacker as a projector based on LLaVA-1.5. This branch does not include patch slicing scheme for TokenPacker-HD.