search

IBM / transition-amr-parser

SoTA Abstract Meaning Representation (AMR) parsing with word-node alignments in Pytorch. Includes checkpoints and other tools such as statistical significance Smatch.
Apache License 2.0
246 stars 48 forks source link

Disconnected Graphs Error #48

Closed Zoher15 closed 1 year ago

Zoher15 commented 1 year ago

Hi @ramon-astudillo ,

I get a very concerning error: image I have no idea how to interpret it, because I have rerun it on the same data multiple times. Half the time I get this error, other times I do not..

Best, Zoher

ramon-astudillo commented 1 year ago

It is odd that its stochastic, we should look into this.

The warning does not mean the generated graph is invalid and actually a small fraction of detached node errors is normal. You can identify them because the AMR graph has :rel edges, signaling that whatever is below that edge could not be attached to the rest of the graph.

Zoher15 commented 1 year ago

@ramon-astudillo Small correction: I was wrong. That error shows up everytime. The error is not stochastic, but when I use AMRs for downstream tasks the result does vary by small amounts. The downstream results are stochastic. My guess is the graphs are not exactly the same every-time I run the parser?

Tverous commented 1 year ago

I found that the parser exhibits stochastic behavior, meaning it can produce varying Penman notations from identical sentences and model weights. Furthermore, when operating the parser in batch mode, it is more likely to generate disconnected graphs.