retrofitting estimate: retrofitting yes/no questions to :polarity amr-unknown

timjogorman commented 7 years ago

This is based on #193 and #185 -- look back at those for more background. I'm just posing an estimate for how much this would cost to actually retrofit.

The rough profile of the :mode interrogative instances:

roughly 2000 other instances that are either at the top or top-like (under a multi-sentence node or under a top-level and, or or contrast-01), of which essentially all are actual yes-no questions.
about 500 with "whether" or "if".
roughly 200 sentences that aren't polar complementizers or top-level yes-no questions. Some of these are "choice" questions; others are just embedded questions or annotation errors.

Proposal

Automatically convert all top-level ":mode interrogative" instances to ":polarity (a / amr-unknown)". (cost: Ulf time. If not automatically doable, then retrofitting cost involves manually changing 2794 AMRs, and might not be worth it. )
Convert all embedding :mode interrogative instances with "whether" or "if" into the truth-value trick, or some other treatment (cost: about 500 quick/easy retrofits; I'm assuming that's too complicated to do automatically)
Manually examine the estimated ~200 remaining instances of :mode interrogative after that. I'd propose doing choice questions using my proposal from #185, but it's not a settled issue. (cost: 200 retrofits).

All told, assuming that people are manually changing complements and that each AMR takes 1 minute, it would require about 15 hours of annotator time to retrofit, assuming that the top-level instances of ":mode interrogative" could be changed automatically. If that needs to be done manually, this would roughly 46 hours of annotator time.

Proposed Treatments

I'm assuming the general proposal from Kevin, with my analyses for "whether" clauses and choice questions; I don't think that these are settled issuse. If those with doubts (I think Nathan and Kira) could produce a separate proposal (ideally re-stating my examples here) we could compare the two.

polarity amr-unknown treatment:

Are you sure you are a conservative?
(s / sure-02
      :ARG0 (y / you)
      :ARG1 (p / person
            :domain y
            :mod (c / conservative))
      :polarity (a / amr-unknown))

Should we have kept interracial marriages illegal since marriage is supposed to be about love?
(c / cause-01
      :ARG0 (s / suppose-02
            :ARG1 (m / marry-01)
            :ARG2 (a / about
                  :op1 (l / love-01)))
      :ARG1 (r / recommend-01
            :ARG1 (k / keep-04
                  :ARG0 (w / we)
                  :ARG1 (m2 / marry-01
                        :mod (i / interracial))
                  :ARG2 (l2 / legal-02 :polarity -))
            :polarity (a2 / amr-unknown)))

truth-value trick:

It did not say if the men had been hanged in prison.
(s / say-01 :polarity -
      :ARG0 (i / it)
      :ARG1 (t / truth-value
            :polarity-of (h / hanging-07
                  :ARG1 (m / man)
                  :location (p / prison))))

Whether inheretance taxes should exist or not is a discussion for another thread.
(d / discuss-01
      :ARG1 (t3 / truth-value
            :polarity-of (r / recommend-01
                  :ARG1 (e / exist-01
                        :ARG1 (t / tax-01
                              :ARG3 (i / inherit-01)))))
      :medium (t2 / thread
            :mod (a / another)))

choice questions

Is it a paradise for the rich or the general public ?
(p / paradise
      :domain (i / it)
      :beneficiary (a / amr-unknown
            :ARG1-of (i2 / include-91
                  :ARG2 (o / or
                        :op1 (r / rich)
                        :op2 (p2 / public
                              :ARG1-of (g / general-02))))))

Was that a US flag or a Confederate flag?
(a / amr-unknown
      :domain (t / that)
      :ARG1-of (i2 / include-91
            :ARG2 (o / or
                  :op1 (f / flag
                        :mod (c / country :wiki "United_States" :name (n2 / name :op1 "US")))
                  :op2 (f2 / flag
                        :mod (s2 / state :wiki "Confederate_States_of_America" :name (n3 / name :op1 "Confederate"))))))

*(can replace include-91 with choice-value-91 if include-91 makes people squeemish)*

nschneid commented 7 years ago

For choice questions, I think include-91 makes me squeamish because of the or layer. There is something noncompositional about allowing each of the or clauses as the possible responses.

Consider:

Who was a teacher or professor that you loved?

(l / love-01 :ARG0 (y / you)
:ARG1 (a / amr-unknown 
:ARG1-of (i / include-91
:ARG2 (o / or 
:op1 (p / person :ARG0-of (t / teach-01))
:op2 (p2 / professor)))))

The expected response would be someone's name. That's a different question from "Was it a teacher or a professor that you loved?" For which I could see something like:

(l / love-01 :ARG0 (y / you)
    :ARG1 (a / amr-unknown 
        :mod (c / choice-value
            :op1 (p / person :ARG0-of (t / teach-01))
            :op2 (p2 / professor))))

Or even:

(l / love-01 :ARG0 (y / you)
    :ARG1 (a / amr-unknown-choice
        :op1 (p / person :ARG0-of (t / teach-01))
        :op2 (p2 / professor)))

nschneid commented 7 years ago

From today's call, it seems we're converged on :polarity amr-unknown, :polarity truth-value, and for choices:

Do you want tea or coffee?

(w / want-01 :ARG0 (y /you)
    :ARG1 (a / amr-choice
        :op1 (t / tea)
        :op2 (c / coffee)))

Was it a teacher or a professor that you loved?

(l / love-01 :ARG0 (y / you)
    :ARG1 (a / amr-choice
        :op1 (p / person :ARG0-of (t / teach-01))
        :op2 (p2 / professor)))

uhermjakob commented 7 years ago

I retrofitted the bulk of yes-no-questions in the AMR release corpus by extending and running my repair-amr script which now automatically updates any top-level :mode interrogative to :polarity (a / amr-unknown)
[or a2/a3/... -- whatever the first unused variable is].

Two exceptions:

No auto-update if the top-level concept is "or", which often occurs in multiple-choice questions: (o / or :mode interrogative :op1 (...) :op2 (...))
No auto-update if the top-level includes both :mode interrogative and :polarity - such as for Isn't he hungry? where we probable want to avoid
(h / hungry :polarity (a / amr-unknown) :polarity - :ARG1 (h2 / he)) Isn't he hungry? is similar to He is hungry, isn't he? which we annotate as: (h / hungry :ARG1 (h2 / he) :ARG1-of (r / request-confirmation-91))

I updated sub-sentences directly under top-level multi-sentence analogously, as well as (most) sub-AMRs directly under top-level and/or/contrast-02 if there is no :op1/:ARG1 (typical for yes-no questions that start with "and/or/but"). Among other things, this avoids updates to AMRs such as (or :op1 (... :mode interrogative ...) :op2 (... :mode interrogative ...)) which often signal a multiple-choice question.

Number of updates performed: 1820 (consistent with Tim's estimate of roughly 2000).

uhermjakob commented 7 years ago

Another tricky case:

Racially insensitive? DF-199-193600-660_5903.5

(s / sensitive-03 :polarity - :mode interrogative   
     :ARG1 (r / race))

amrisi / amr-guidelines