Question about your mlm mask

Albert-Ma / PROP

WSDM'2021, PROP and SIGIR'2021,B-PROP

Apache License 2.0

110 stars 18 forks source link

Closed zyznull closed 3 years ago

zyznull commented 3 years ago

I notice that in the pretrainng process, the mlm words do not be masked? Is this a bug of this version code?

Albert-Ma commented 3 years ago

No, we preprocess the data including masking words and constructing the ROP task before the pretraining. See this