isi-nlp / DiplomacyAMR

AMR-related aspects of Diplomacy ALLAN project
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Build-01 :location #27

Closed saramosher888 closed 2 years ago

saramosher888 commented 2 years ago

Here is another issue with the guidelines / checker that is causing consistency problems in annotation. For move-01, we are modifying the unit with the location and the country but leaving :arg0 blank. Example: "Move the English army in Paris to Brest." image

Contrast this with build-01. Until our Github conversation last week, I thought we were handling move-01 and build-01 in the same way, but now I understand that this is not the case. From the beginning, it was made very clear to me that we should be modifying the unit with the :location, like this: Example: "Build an English army in Paris"
image

But now the checker is finding an error if I modify the unit with the location. Instead, it wants me to modify build-01 directly, like this: image

My concern is that we are not going to have consistency in the annotation if there is this kind of inconsistency in the basic structure of the guidelines. We need to modify the unit in the same way regardless of which concept we happen to be annotating at that particular moment. Otherwise we're setting up a process that's prone to constant errors because the same basic type of information needs to be entered in different places every time.

What's the reason for modifying build-01 with :location directly, instead of adding the location to the unit as we do with move-01?

uhermjakob commented 2 years ago

So yes, build-01 is annotated differently from move-01 and friends. The main difference/rationale is that the military unit owned by a country and located at a location does not exist at the beginning of the build process, only as a result. Annotators always intuitively annotated as it is in the guidelines now. It would be slightly weird to say "The Russian army in Warsaw is being built." We had a lengthy discussion, and we did consider annotating build-01 resultatively.

saramosher888 commented 2 years ago

I'm thinking about how to make this more consistent. There are also other times when we use "unit". For example, we've sometimes been using it for transport-01. (transport-01 :instrument unit). Ideally, I'd like to be able to describe the unit in the same way across different concepts without generating errors. It would simplify things for us if we could attach the location to the unit. Would this cause a problem?

uhermjakob commented 2 years ago

When talking about an existing military unit, its location and nationality are identifying characteristics. That is a major reason why we annotate the location and nationality under army/fleet/unit in those cases. The build-01 frame is different in that the location and nationality are not identifying characteristics of the military unit before or during the building process. As mentioned before, we found that annotators consistently annotated build-01 as currently documented in the guidelines. The AMR Checker will catch non-canonical annotations in this respect. Annotators are encouraged to use the AMR Checker frequently (say at least after about 10-20 AMRs), which will help them correct any deviations from our guidelines (and learn to follow our guidelines) sooner than later.