Still not supporting for ChatGLM3 maybe

[fjy01]

System Info

trl 0.12.1 transformers 4.46.2

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder
• [X] My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModel, BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments, AutoModelForCausalLM
import torch
import transformers
from peft import PeftModel
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead

model = AutoModelForCausalLMWithValueHead.from_pretrained("THUDM/chatglm3-6b-128k", trust_remote_code=True)

outputs:

Traceback (most recent call last):
  File "/home/fjy/folders/ERNIE2.0/finetune_chatglm6b.py", line 16, in <module>
    model = AutoModelForCausalLMWithValueHead.from_pretrained("THUDM/chatglm3-6b-128k", trust_remote_code=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fjy/anaconda3/envs/env/lib/python3.12/site-packages/trl/models/modeling_base.py", line 233, in from_pretrained
    model = cls(pretrained_model, **trl_model_args)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fjy/anaconda3/envs/env/lib/python3.12/site-packages/trl/models/modeling_value_head.py", line 107, in __init__
    raise ValueError("The model does not have a language model head, please use a model that has one.")
ValueError: The model does not have a language model head, please use a model that has one.

Expected behavior

It's shown that ChatGLM3 doesnot have a language model head. Appearly it has.

Checklist

• [X] I have checked that my issue isn't already filed (see open issues)
• [X] I have included my system information
• [X] Any code provided is minimal, complete, and reproducible (more on MREs)
• [X] Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• [X] Any traceback provided is complete