Problems to run

[ansiiso]

1. Variable $BASE_WDIR is not defined. https://github.com/idio/wiki2vec/blob/master/prepare.sh#L32
2. Got "MemoryError" while trying to build model and found this post: https://github.com/piskvorky/gensim/issues/293. Have you solved the issue? How large the memory it requires to run?

[dav009]

1. Are you running prepare.sh on ubuntu 14.04 ?
2. Nope. You can get around it in two ways: (i) Get an instance with more RAM, (ii) increase min_count

A few more questions:

• Are you using this repo to process a wiki? if so what language?

[ansiiso]

1. Yes, I am using the script on ubuntu 14.04.
2. I am processing English wiki with 2xlarge ec2 instance: 8 vCpu, 30 GB

[dav009]

• Just for the record I generated a big model (1000Dimension, I think 10Cbow) and I had similar errors on a 30G instance, this was due to the memory error on Gensim. So I had to bump it to a 60G instance.
• How many dimensions are you generating?, I think I will upload the generated models soons in case you are interested
• Also could you create a PR for the BASE_WDIR issue :)

[ansiiso]

• I am using setting from README.md: wiki2vec.sh corpus output 50 500 10.
• It will be very helpfully if you could share generated models from English Wiki. Thank you!

[dav009]

If you are bounded to the 30G instance, You could try building a model with : 100 300 10. 300 dimensions, 100 min count.

[dav009]

@ansiiso check the readme for instructions on how to get the english model via torrent. It is around 8G and there are few seeds so it should take a while to download.

[dav009]

@ansiiso closing this issue as the memory limitation is due to gensim implementation / aws instance size.