Can we use the ChatGLM to train on the masked language modeling task, just like BERT?
Like "[MASK] is the Capital of U.S.", given the correct answer Washington, can we use this to train ChatGLM?
Steps To Reproduce
I don't know
Environment
- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :
Is there an existing issue for this?
Current Behavior
I can't find anything about this.
Expected Behavior
Can we use the ChatGLM to train on the masked language modeling task, just like BERT? Like "[MASK] is the Capital of U.S.", given the correct answer Washington, can we use this to train ChatGLM?
Steps To Reproduce
Environment
Anything else?
No response