Open agemagician opened 3 years ago
I think it should be possible to train like RoBERTa. If NSP is set to false, next_sentence_labels
is just dummy. You can just fill it to be all 0 or 1. It isn't used for any gradient update. For example in the simple TFDS based example in bigbird/pretrain/run_pretraining.py
we train with a single sequence per example without NSP and next_sentence_labels
is set to all 0 in Line 193.
Thanks a lot @manzilz for your reply.
So, we need to disable "preprocessed_data" and perform masking on the fly.
Could you please give us a concrete example for using on the fly masking with a local dataset for pre-traininng ?
From the code I can see we have to either use "tfds" online hub or tfrecrods which is already preprocessed.
Assuming I have a local folder with several text files like "data/data_*.txt", what is the correct commands that we should use to train Roberta like model ? or we must have the data on the tfds online hub ?
We used tfrecords or TFDS for efficiency as reading large corpus of text files is not as efficient when training at large scale. I think it should easy to modify bigbird/pretrain/run_pretraining.py
to read from local text files instead of TFDS. Instead of getting text from TFDS in line 297 we can insert code for generating a tf.Dataset from local text files. Then do_masking should handle on the fly masking and rest should work.
Thanks again for your explanation.
Could you please provide the code for generating the tfrecords ?
Unfortunately, if we used Bert tfrecords generator, it will combine two sequences per sample.
Is it possible to share either:
This will be super useful for training a new custom dataset.
Hello,
First, congratulations for your work.
Second, from what I have discovered so far, you only allow Bert like training and not Roberta training. Even if the NSP is set to false, still your script requires the "next_sentence_labels" field which is generated by Bert script.
My question is: How can we generator and train a model like Roberta, where there is only a single sequence per example without NSP.
@manzilz @ppham27 your feedback is highly appreciated. Thanks in advance for your reply.