search

microsoft / dp-transformers

Differentially-private transformers using HuggingFace and Opacus
MIT License
109 stars 20 forks source link

`DataCollatorForPrivateCausalLanguageModeling` hardcoded to `mlm=False` #42

Open madaanpulkit opened 5 months ago

madaanpulkit commented 5 months ago

Why is the mlm parameter hardcoded to False in DataCollatorForPrivateCausalLanguageModeling's implementation? I intend to use the DPCallback for Masked Language Modeling. Is it unsafe to turn it to True?

dp_utils.py#L99

huseyinatahaninan commented 5 months ago

It's totally safe to turn it to True. It's just in our examples we generally worked with unidirectional language models and we constructed this DataCollator for "Causal Language Modeling", hence mlm parameter is hardcoded to False. But you can totally use mlm=True for Masked Language Modeling. You may not even need this particular DataCollator if you don't have the similar issue with position_ids. I guess we could have been more comprehensive by calling it DataCollatorForPrivateLanguageModeling and let the user input the parameter mlm :)