Closed dhimasyoga16 closed 4 years ago
hi @dhimasyoga16 thank you for your interest in this repo. I am not sure what the question is asking.
Are you asking for how to fine-tune the bert-multilingual
model? For that, you need to check the huggingface model to see how to continue training bert-multilingual
with the mask language modeling objectives. See https://github.com/huggingface/transformers/tree/master/examples/language-modeling.
If you have already fine-tuned bert-multilingual
, you can feed in the model path to --model
when calling the score
function.
Feel free to follow up with more questions.
Hi, thankyou so much for the quick assist and the answer. I also feel sorry for my hard-to-understand question, my bad.
Can i ask one more question?
I've done feature extracting by running the extract_features.py
and it generates 17.3GB json file. Can i use that json file as my model?
Because i want the BERTScore to analyze Indonesian sentences/texts better.
Thankyou so much once again :)
hi @dhimasyoga16 which extract_features.py
file are you talking about? is it in this repo?
if you have precomputed the features, you can modify the code (https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L253) to load the features instead of computing them again.
Hi, sorry for the inactivity of this issue. Refer to this link : https://github.com/huggingface/transformers/tree/master/examples/language-modeling
How can i fine-tune the bert multilingual model in Indonesian language? Can i use a wikipedia dump file as a corpus?
hi @dhimasyoga16 this is a question that is better posed to the huggingface repo. Hopefully they will have a detailed instruction.
We are not really experts on this topic.
Hi, i've successfully created my language model using huggingface transformers.
When i'm trying to do a testing (using my own model of course), why does the --num_layers
affect the score?
For example, when i'm using --num_layers 2
it gave me lower score rather than when i'm using --num_layers 6
.
Any detailed explanation about this?
And which --num_layers
that will give better scoring accuracy?
Sorry for so much questions, i'm new to NLP.
hi @dhimasyoga16 please see our paper for the effect of using different --num_layers
. basically, this argument controls which pre-trained layer of representations you are using. It is hard to say which --num_layers
would work best for you application without seeing any validation data from our side.
I have a wikipedia dump file as my corpus (which is in Indonesian, i've extract it and convert it to .txt) How can i train my corpus file with
bert multilingual cased
(fine tune) with BERTScore so i can have my own model for specific task such as sentence similarity or automated short answer scoring?Or maybe i should do this with the original BERT? Thankyou so much in advance.