Open maxaalexeeva opened 2 years ago
@IkeKobby fyi
@MihaiSurdeanu Allegra needed it at some point to distinguish between events/sentences related to harvesting, planting, etc. Not sure how crucial it is now.
On planting areas: to handle the ambiguous phrases, we could extract generic "area" in the grammar, and then have an action that checks if some meaningful keywords appear anywhere in the sentence, e.g., "sow", "cultivate", etc.'
ok
I can imagine wanting some other algorithm entirely, but for getProcess() it should be easy enough to count how many matches there are for each key and use the key of the maximum value. The result would be dependent on the ordering only in the case of ties.
import scala.collection.immutable.ListMap
val processToLemmas = ListMap(
"planting" -> Seq("plant", "sow", "cover", "cultivate", "grow"),
"harvesting" -> Seq("harvest", "yield"),
"credit" -> Seq("credit", "finance", "value", "correspond"),
"natural_disaster" -> Seq("flood", "bird", "attack")
)
def getProcess(mention: Mention): String = {
val lemmas = mention.sentenceObj.lemmas.get
val process = processToLemmas
.mapValues(_.count(lemmas.contains))
.maxBy(_._2)
._1
process
}
@kwalcock I like that! Thank you!
There are two versions of getProcess above. I guess the algorithm is in flux. In the top one there is a helper class that I decided was not necessary and I also forgot about ListMap. The code could look more like the bottom one.
import scala.collection.immutable.ListMap
// These are prioritized highest to lowest.
val processToLemmas = ListMap(
"planting" -> Set("plant", "sow", "cultivate", "cultivation", "grow"),
"harvesting" -> Set("harvest", "yield"),
"credit" -> Set("credit", "finance", "value"),
"irrigation" -> Set("irrigation", "irrigate"),
"weeds" -> Set("weed"),
"natural_disaster" -> Set("flood", "bird", "attack")
)
def getProcess(mention: Mention): String = {
val sentenceLemmas = mention.sentenceObj.lemmas.get
val process = processToLemmas
.find { case (_, processLemmas) =>
sentenceLemmas.exists(processLemmas)
}
.map { case (process, _) => process }
.getOrElse("UNK")
process
}
@kwalcock thanks!
These are two issues related to each other.
[ ] we have a rudimentary process "classifier" here which is right now based on string match, which does not account for conflicting key terms within one sentence (but this has also not been crucial/used by anyone in a while). Any suggestions on how to improve that?
[ ] We are trying to extract planting areas (
PlantingArea
). Planting areas are not always explicitly marked (they are just referred to asarea
s). Other types of areas, on the other hand, frequently have indicators, e.g., irrigation and weed areas below.Four scores , corresponding to four classes of surface area coverage by weeds , were distinguished : 0 for none ; 1 for weak ( less than 10 % of surface area coverage by weeds ) ; 2 for strong ( between 10 and 30 % ) ; 3 for very strong ( more than 30 % of surface area coverage by weeds ) .
This type of irrigation scheme ( or perimeter ) , with an area of below 50 ha and cultivated by farmers from a single village , covers about 25 % of the irrigated area on the two banks of the Senegal River ( SAED , 1997 ; SONADER , 1998 ) .
[ ] make a separate class for processing lemmas as recommended by @kwalcock: