search

slavpetrov / berkeleyparser

Automatically exported from code.google.com/p/berkeleyparser
GNU General Public License v2.0
180 stars 48 forks source link

Documentation of file formats #9

Open t-wissmann opened 4 years ago

t-wissmann commented 4 years ago

Where can I find documentation of the file formats used? Unfortunately I neither can find one for the .gr files in the repo nor for the file formats generated by WriteGrammarToTextFile such as .grammar (as described in README).

Though I can guess most of what's in the .grammar files I'm still a bit puzzled. I have invoked by the command

java -cp BerkeleyParser-1.7.jar edu/berkeley/nlp/PCFGLA/WriteGrammarToTextFile arb_sm5.gr arb_sm5

and in the content of arb_sm5.grammar I'm wondering:

(I also couldn't find any notes on the file format in the publications COLING-ACL 2006 and HLT_NAACL 2007 that are mentioned in the README).

The reason I am asking is that I'm considering supporting .gr or .grammar input files in an own project CoPaR.

t-wissmann commented 4 years ago

Regarding the .gr, I've noticed from ParserData.Load (ParserData.java line 104f) that .gr is a gzipped java object stream.

t-wissmann commented 4 years ago

Some transitions starting at ROOT_0 are duplicated, e.g.:

ROOT_0 -> ROOT_0 1.0
ROOT_0 -> ROOT_0 1.0

What does this mean? Can these duplicates be ignored or do the weights 1.0 sum up to 2.0 such that the above transitions are equivalent to the following?

ROOT_0 -> ROOT_0 2.0