search

feralvam / easse

Easier Automatic Sentence Simplification Evaluation
GNU General Public License v3.0
157 stars 36 forks source link

resourceKilled #94

Open RitaHjz opened 2 years ago

RitaHjz commented 2 years ago

Dear Fernando, Thank you for developing Easse tool. It helps me a lot. However, I’m trying to use SAMSA metric on my output but it fails to compute it. Could you help me to solve it? I tried to download SAMSA but the tool suffers from insufficient info about how to use it and I didn’t understand the code. Here is the error message:

rita@rita-VirtualBox:~/easse$ easse evaluate -t turkcorpus_test -m 'samsa' -q < easse/resources/data/system_outputs/turkcorpus/test/R Warning: SAMSA metric is long to compute (120 sentences ~ 4min), disable it if you need fast evaluation. Loading spaCy model 'en_core_web_md'... Done (76.791s). Loading from '/home/rita/.local/lib/python3.8/site-packages/easse/resources/tools/ucca-bilstm-1.3.10/models/ucca-bilstm.json'. [dynet] random seed: 1 [dynet] allocating memory: 512MB [dynet] memory allocation done. [dynet] 2.1.2 Loading from '/home/rita/.local/lib/python3.8/site-packages/easse/resources/tools/ucca-bilstm-1.3.10/models/ucca-bilstm.enum'... Done (0.295s). Loading model from '/home/rita/.local/lib/python3.8/site-packages/easse/resourceKilled

Thanks in advance!

louismartin commented 2 years ago

Hi @RitaHjz, Thanks for opening this issue. To be fair SAMSA is pretty hard and long to use and we have not implemented the original logic ourselves: https://github.com/eliorsulem/SAMSA

I am not sure what causes your process to be killed, it probably happens outside of the main python thread. It seems to happen when loading a model, maybe it's a memory error?

RitaHjz commented 2 years ago

Hi @louismartin, Thanks for your quick answer. I tried with few sentences but I have always the same problem :(

feralvam commented 2 years ago

Hi @RitaHjz,

I ran the following command in a fresh installation of EASSE and it worked fine:

easse evaluate -t turkcorpus_test -m 'samsa' -q < easse/resources/data/system_outputs/turkcorpus/test/RM+EX

As Louis mentioned, it seems to be related to lack of memory space when loading the ucca-bilstm model. As such, the number of sentences in the file does not influence there. Have you tried running it in colab, for instance?