dshefman1
commented
4 years ago I was able to solve the problem by inferring the format of both vectors.txt and vocab.txt from the 05_export.py script _get_shape and read_vocab functions. Also, in the 05_export.py script I had to change line 15 to first_line = next(file_).replace('\ufeff','').split() because of Windows' UTF-8 BOM signature included at the beginning of UTF-8 text docs.
Here is a complete Windows 10 compatible 04_fasttext_train_vectors.py script:
from pathlib import Path
from wasabi import msg
import fasttext
in_dir = "./corpus_parsed3"
out_dir = "./fasttext_model3"
n_threads = 26
min_count = 50
vector_size = 300
verbose = 2
input_path = Path(in_dir)
output_path = Path(out_dir)
if not input_path.exists() or not input_path.is_dir():
msg.fail("Not a valid input directory", in_dir, exits=1)
if not output_path.exists():
output_path.mkdir(parents=True)
msg.good(f"Created output directory {out_dir}")
output_file = output_path / f"vectors_w2v_{vector_size}dim.bin"
# fastText expects only one input file and only reads from disk and not
# stdin, so we need to create a temporary file that concatenates the inputs
tmp_path = input_path / "s2v_input.tmp"
input_files = [p for p in input_path.iterdir() if p.suffix == ".s2v"]
if not input_files:
msg.fail("Input directory contains no .s2v files", in_dir, exits=1)
with tmp_path.open("a", encoding="utf8") as tmp_file:
for input_file in input_files:
with input_file.open("r", encoding="utf-8") as f:
tmp_file.write(f.read())
msg.info("Created temporary merged input file", tmp_path)
sense2vec_model = fasttext.train_unsupervised(in_dir+"/s2v_input.tmp", thread=n_threads, epoch=5, dim=vector_size, minn=0, maxn=0, minCount=min_count, verbose=verbose)
# sense2vec_model.save_model(out_dir+f"/vectors_w2v_{vector_size}dim.bin")
tmp_path.unlink()
msg.good("Deleted temporary input file", tmp_path)
words, freqs = sense2vec_model.get_words(include_freq=True)
with open("./fasttext_model3/vocab.txt", 'w', encoding='utf-8') as f:
for i in range(len(words)):
f.write(words[i] + " " + str(freqs[i]) + " word\n")
# https://stackoverflow.com/questions/58337469/how-to-save-fasttext-model-in-vec-format
# get all words from model
words = sense2vec_model.get_words()
# print(str(len(words)) + " " + str(sense2vec_model.get_dimension()))
# line by line, you append vectors to VEC file
with open("./fasttext_model3/vectors.txt", 'w', encoding='utf-8') as file_out:
file_out.write(str(len(words)) + " " + str(sense2vec_model.get_dimension())+'\n')
for w in words:
v = sense2vec_model.get_word_vector(w)
vstr = ""
for vi in v:
vstr += " " + str(vi)
try:
file_out.write(w + vstr+'\n')
except:
pass
svlandeg
Z-e-e
The two os.system(cmd) portions of the 04_fasttext_train_vectors.py script on lines 61-67 and lines 75-81 do not work for Windows users. So, I re-wrote lines 61-67 using the FastText Word representations documentation. However, lines 75-81 are proving more difficult to rewrite because I don't know the structure of the vocab_file output file created with lines 76-78, included below. The 3rd-to-last line of my code below uses the save_model function to save the model to a binary file for later loading as shown here. However, this is not the input file format expected in 05_export.py. Could you please provide a sample of what the vocab_file output file looks like? Or, better yet, do you have any suggestions for how to replace lines 75-78 that doesn't involve using os.system, CLI code, or the fasttext_bin.
Lines 76-78 of 04_fasttext_train_vectors.py:
Here is the code I used in place of 04_fasttext_train_vectors.py to make it Windows compatible: