What're the modifications in `llava/train/transformers_replace`?

[ys-zong]

Hi, thanks for the nice work! I wonder what are the main modifications in llava/train/transformers_replace compared to the original implementation in transformers==4.31.0, as specified in the pyproject.toml. Also, in environment_setup.sh, transformers==4.36.2 is installed:

pip install git+https://github.com/huggingface/transformers@v4.36.2

I wonder why we want to install different versions of transformers?

If I want to use a higher version of transformers, e.g. 4.38, are there changes needed for the files in this folder? Many thanks!

[Lyken17]

We have manually changed some original implementations to better support grouping strategy and flash attn, and recommend every VILA user to do so.

Though our codebase should work with higher version transformer, we haven't tested throughfully thus cannot promise anything. Please use v4.36.2 for reproducement.

[wusize]

I found "transformers_version": "4.38.1", in the config.json of VILA-2.7B. Which version of transformers should we use to run VILA-2.7B? By the way, is there any plan to allow loading VILA models using transformers API only without relying on the current repo?

[weichow23]

I noticed that you mainly manually implemented class LlamaForCausalLM(LlamaPreTrainedModel) in I noticed that you mainly manually implemented class LlamaForCausalLM(LlamaPreTrainedModel) in replace. however, in the code (either train or infer), your implemented class is not be used. What's wrong with it? Thanks for your reply

[amitbcp]

@Lyken17 : Could you please share some details for this ?

[Lyken17]

please use 4.36.2 for now, we will upgrade to 4.37 in next release :)

[JBurtn]

Also why manually replace the files? Why not use monkeypatching, or reregister your versions with transformers using exist_ok=True?