Closed stribizhev closed 4 years ago
Our models are not trained with the internal sentencepiece segmentation and but we train the sentencepiece model before starting the MT training procedures. That means, that the data you train on must be segmented with that model before you can start MarianNMT. So, run spm_encode with the provided model and continue training with the marianNMT vocabulary in the json file. Do not add the spm-models to the call. That should work. Let e know if it doesn't.
Is this still an issue?
Yes, thank you, the issue is resolved.
@stribizhev : Have you been successful with improving ENSK model? (I am sorry for this, but I did not find other way how to contact stribizhev...)
@stribizhev : Have you been successful with improving ENSK model? (I am sorry for this, but I did not find other way how to contact stribizhev...)
I have not been using these models for ages, I am involved in fine-tuning our own models. As for the issue in this ticket, the problem was that I did not know we can include JSON vocabulary files, I have always been using SPM files.
Problem: incremental training using Marian 1.9 did not succeed using additional bilingual data and the ENSK model and SPM model downloaded from this repo.
Exception stack:
Steps:
"${marian_path}/marian" -c "${cfg_file}" --model "${fldr}/opus.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz" --no-restore-corpus
, using source.spm and target.spm as the vocabulary files.Error: Requested shape shape=32000x512 size=16384000 for existing parameter 'Wemb' does not match original shape shape=60024x512 size=30732288
wc -l source.spm
returned 69880, andwc -l target.spm
returned 69359. There's also a file opus.spm32k-spm32k.vocab.yml*,wc -l opus.spm32k-spm32k.vocab.yml
returns 60023.opus.spm32k-spm32k.vocab.yml
as the source and target vocabularies, but this produced the following error:Error: Detokenizing BLEU validator expects the target vocabulary to be SentencePieceVocab or FactoredVocab. Current vocabulary type is DefaultVocab
I suspect the SPM shipped with the ENSK model was trained on a different corpus.
Thank you for any hints how to solve the problem.