Closed MinionAttack closed 3 years ago
@MinionAttack Hi, it occurs because fasttext embeddings are very huge. You can refer to #36 to solve the issue.
Thanks for the reply, I'm going to do a fair comparison and I've seen that the other parser I'm using limits the alphabet to 200K embeddings, so I've switched:
@classmethod
def load(cls, path, unk=None):
with open(path, 'r') as f:
lines = [line for line in f]
splits = [line.split() for line in lines]
tokens, vectors = zip(*[(s[0], list(map(float, s[1:])))
for s in splits])
return cls(tokens, vectors, unk=unk)
to
@classmethod
def load(cls, path, unk=None):
MAX_VOCABULARY_SIZE = 200000
with open(path, 'r') as f:
lines = [line for line in f]
if len(lines) > MAX_VOCABULARY_SIZE:
lines = lines[:MAX_VOCABULARY_SIZE]
splits = [line.split() for line in lines]
tokens, vectors = zip(*[(s[0], list(map(float, s[1:])))
for s in splits])
return cls(tokens, vectors, unk=unk)
And now the RAM consumption is normal.
Hi,
I am trying to test SuPar in several languages and after running several tests with 100 dimension embeddings, I am now trying to repeat the experiments with 300 dimension embeddings.
The problem is that SuPar, as soon as it starts, in the "INFO Building the fields" phase, starts to occupy the RAM memory in a crazy way. My test computer has 64 GB of RAM and a SWAP partition of 122 GB, so in less than 4 minutes I had all the RAM occupied and the SWAP partition was 30 GB occupied and going up.
I am using the Universal Dependency files for English (EWT) and for the embeddings I'm using the FASTTEXT model for English (after deleting the first line by hand).
This is the configuration file:
And this is the command to launch the parser (batch file):
And the parser, just before the "INFO Building the fields" message, prints:
Regards.