Trained my model using phrase2vec.py but now I want to test using that model. How?

[monipip3]

I used my own corpus using this command in the Read Me replacing the corpus_example with my own, it created a new model called model_example in the same folder as the pre trained embeddings. python phrase2vec.py --corpus=data/corpus_example --model_name=model_example

I want to then use this new model to test. How do I do that? I thought I would run

from gensim.models import Word2Vec w2v_model = Word2Vec.load("mat2vec/training/models/model_example")

but I got an error saying No such file or directory: '/Users/monicapuerto/Desktop/Github/mat2vec/mat2vec/processing/models/phraser.pkl'

When that file does indeed exist. Why does the new model depend on this file? How do I utilize the model I just trained?

[monipip3]

so now I ran process.py after and then from gensim.models import Word2Vec w2v_model = Word2Vec.load("mat2vec/training/models/model_example") and it seemed to work because I did not get an error message.

I used a word from the corpus I used to train on in the most_similar function and it said that word is not in the vocabulary. How is that possible? Also getting a lot of stop words as most similar to most words in my corpus.

[ViktoriiaBaib]

To have the word in the vocabulary, it should appear more than 5 times in your corpus by default. Option 1: expand your corpus - add more articles containing the desired word. Option 2: change this parameter using the flag "--min_count". For example, if you want to build a vocabulary containing all words run the following: python phrase2vec.py --corpus=data/your_corpus --model_name=your_model --min_count=1

[ViktoriiaBaib]

Make sure that all the paths are correct and you do the following in the given order:

1. Do all the steps from the "Set up" section of the README
2. Go to mat2vec/mat2vec/training/data and create 'my_corpus' file containing abstracts (for the reference look at the content of 'corpus_example')
3. Go to mat2vec/mat2vec/training and build the vocabulary for your corpus running python phrase2vec.py --corpus=data/my_corpus --model_name=my_model. This command takes existing file 'data/my_corpus' and generate new file 'models/my_model'.
4. Go to mat2vec, run python interactive shell and try some of the steps from the "Processing" section of the README but importing 'my_model', like: w2v_my_model = Word2Vec.load("mat2vec/training/models/my_model")

[vtshitoyan]

To add to this, the processing needs to be done before creating the corpus file. The corpus file should contain pre-processed, tokenized abstracts, one per line. Tokens should be separated by spaces. As to how to test the model, it depends what you want to test. Some examples are in README, also check out some of the other issues in this repo which might have helpful tips.

[hasan-sayeed]

Is it possible to load the "pretrained model" and then train it again on my custom corpus? I've tried using w2v_model = Word2Vec.load("mat2vec/training/models/pretrained_embeddings") and then w2v_model.train('corpus_example', total_examples=w2v_model.corpus_count, epochs=20). But after the training I tried to find the vector for a word from my corpus and it said the word is not in the vocabulary. Also all the vectors stays the same as the previous model. What am I doing wrong?

[jdagdelen]

In theory you should be able to resume training of a gensim word2vec model. See https://datascience.stackexchange.com/questions/10695/how-to-initialize-a-new-word2vec-model-with-pre-trained-model-weights

On Thu, Aug 19, 2021 at 3:48 PM Hasan Muhammad Sayeed < @.***> wrote:

Is it possible to load the "pretrained model" and then train it again on my custom corpus? I've tried using w2v_model = Word2Vec.load("mat2vec/training/models/pretrained_embeddings") and then w2v_model.train('corpus_example', total_examples=w2v_model.corpus_count, epochs=20). But after the training I tried to find the vector for a word from my corpus and it said the word is not in the vocabulary. Also all the vectors stays the same as the previous model. What can I do?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/materialsintelligence/mat2vec/issues/19#issuecomment-902300949, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFODYJIOQTLVQWYWXS7UADT5WC2JANCNFSM4KNDET5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .