Check EOS token on FastChat models

[natolambert]

TLDR:

• HF tokenizer files add EOS token at the end of instructions https://huggingface.co/HuggingFaceH4/zephyr-7b-beta/blob/b70e0c9a2d9e14bd1e812d3c398e5f313e93b473/tokenizer_config.json#L34
• DPO code manually adds EOS token https://github.com/huggingface/trl/blob/6c2f829bb7408660b0e581cde56fbff0980b9d7b/trl/trainer/dpo_trainer.py#L668

Seems like FastChat models are not getting an EOS token. Could effect the non-DPO models that are using FastChat chat templates minorly

[natolambert]

Solution for the relevant models (will need more filtering code)

    ############################
    # Tokenization settings & dataset preparation
    ############################
    # set pad token to eos token if not set
    if reward_pipe.tokenizer.pad_token_id is None:
        reward_pipe.model.config.pad_token_id = reward_pipe.tokenizer.unk_token_id
        reward_pipe.tokenizer.pad_token_id = reward_pipe.tokenizer.unk_token_id
    # make tokenizer add eos
    reward_pipe.tokenizer.add_eos_token = True