Crash When Running score.sh

bjascob commented 11 months ago

I tried this and it crashed on one dataset but worked on another. Here's the test set that crashed test-gold.txt and test-pred.txt.

python3 -m smatchpp  -a $GOLD \
                     -b $PRED \
                     -solver ilp \
                     -syntactic_standardization dereify \
                     -score_dimension main \
                     -score_type micromacro \
                     -log_level 20 \
                     --bootstrap \

With the traceback..

...
2023-11-28 16:16:00,233 - __main__ - INFO - bindings - graph pairs processed: 1500; time for last 100 pairs: 2.1990966796875
2023-11-28 16:16:03,175 - __main__ - INFO - bindings - graph pairs processed: 1600; time for last 100 pairs: 2.9419069290161133
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/bjascob/.local/lib/python3.10/site-packages/smatchpp/__main__.py", line 162, in <module>
    match_dict, status = SMATCHPP.process_corpus(amrs, amrs2)
  File "/home/bjascob/.local/lib/python3.10/site-packages/smatchpp/bindings.py", line 128, in process_corpus
    match, tmpstatus, _ = self.process_pair(a, amrs2[i])
  File "/home/bjascob/.local/lib/python3.10/site-packages/smatchpp/bindings.py", line 71, in process_pair
    g2 = self.graph_reader.string2graph(string_g2)
  File "/home/bjascob/.local/lib/python3.10/site-packages/smatchpp/interfaces.py", line 60, in string2graph
    triples = self._string2graph(string)
  File "/home/bjascob/.local/lib/python3.10/site-packages/smatchpp/data_helpers.py", line 85, in _string2graph
    triple = (tmpsrc[nested_level], tmprel[nested_level], stringtok)     
KeyError: 5

Looks like you've got an array indexing error.

I don't need this fixed, but since I spotted the issue I thought I'd pass it along.

flipz357 commented 11 months ago

Hi, thanks, I'm always looking for such examples. The "philosophy" of Smatch++ is to treat all parser fairly and rescue as many triples as possible. However, the specific AMR that causes this problem seems really broken (see at "op2")

(o / offer-01
      :ARG0 (c / country
            :name (n / name
                  :op1 "France"))
      :ARG1 (a / and
            :op1 (a2 / aircraft-type
                  :name (n2 / name
                        :op1 "Mirage"
                        :op2 " ) :poss country_0 :ARG1-of ( upgrade-01 ) ) :op2 ( transfer-01 :ARG0 country_0 :ARG1 ( technology ) :ARG2 ( country_1 :name ( name_2 :op1 "))))

Still the script shouldn't crash. I suppose in these cases we could still rescue all triples up unto :op1 "Mirage", and issue a warning. What dou you think?

flipz357 commented 11 months ago

Update, it should be fixed with commit ff4e5d6

I.e., there won't be a runtime error anymore but a warning is issued that the graph is broken.

flipz357 commented 11 months ago

Closing this issue, thanks @bjascob for filing the issue and providing examples.

flipz357 / smatchpp

Crash When Running score.sh #9