search

coli-saar / am-parser

Modular implementation of an AM dependency parser in AllenNLP.
Apache License 2.0
30 stars 10 forks source link

make concatenation of AMR files consistent in preprocessing #79

Open jgroschwitz opened 4 years ago

jgroschwitz commented 4 years ago

Each AMR dataset (e.g. the dev set of LDC2017T10) consists of multiple files. In preprocessing, these files are concatenated in arbitrary order (determined by the OS). This order is not always consistent (I just had a case where one file from a first preprocessing run and a file from a different preprocessing run had different orders, leading to problems).

The concatenation should instead be consistent over different runs, e.g. by lexical ordering of filenames.

jgroschwitz commented 4 years ago

Also while at it, make sure to exclude hidden files from iteration