The idea is to run the sentences A and B from the SICK.txt corpus through FreeLing, and use its tokenization to feed SynaxNet to obtain the UD dependencies. We generate a CONLL file with the SynaxNet output, augmented with the lemmas and senses obtained from FreeLing.
We needed to disable FreeLing's MWE module.
We also add the original POS tag from Freeling to the MISC field so we can compare to SyntaxNet's. The POS tag from Freeling is relevant to understand the sense selected.
To reproduce:
Download SICK.txt from here: http://clic.cimec.unitn.it/composes/sick.html
Install Freeling 4.0 from:
http://nlp.cs.upc.edu/freeling/
https://github.com/tensorflow/models/tree/master/syntaxnet
https://github.com/tensorflow/models/blob/master/syntaxnet/universal.md
https://github.com/ontologyportal/sumo
Fix the paths in env.sh, process-freeling.py, and parse.sh to point to your local installation paths.
Run the stats.sh script
See README in pairs/ and derivated/ for more information.