CogComp / cogcomp-nlp

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
http://nlp.cogcomp.org/
Other
470 stars 142 forks source link

Sentence splitter mistake #492

Open danyaljj opened 7 years ago

danyaljj commented 7 years ago

For this input:

On each chromosome pair are smaller bodies (genes) which control all aspects of a bird, its colour, shape, bone structure, length of feather, size etc. The feather is a structure which is unique to birds.interpreted them as two paired rows of structures that are anatomically very much like feathers, and which are in positions like those of birds.

I expect two sentences, while it gives me only one sentence: http://austen.cs.illinois.edu:8080/annotate?text=On%20each%20chromosome%20pair%20are%20smaller%20bodies%20(genes)%20which%20control%20all%20aspects%20of%20a%20bird,%20its%20colour,%20shape,%20bone%20structure,%20length%20of%20feather,%20size%20etc.%20The%20feather%20is%20a%20structure%20which%20is%20unique%20to%20birds.interpreted%20them%20as%20two%20paired%20rows%20of%20structures%20that%20are%20anatomically%20very%20much%20like%20feathers,%20and%20which%20are%20in%20positions%20like%20those%20of%20birds.&views=

danyaljj commented 7 years ago

It mistakenly splits on "Lord.":

screen shot 2017-10-21 at 7 55 38 pm
mssammon commented 7 years ago

But that's a period after the first "Lord" ? And that is the first word in the text? If so, what other behavior would you expect?

danyaljj commented 6 years ago

The previous example: agreed wiith Mark.

This example uses ; as sentence splitter. Wondering if we can do anything about this:

He increased the faculty by more than 500 professors ; the academic quality of the student body has improved dramatically , with the average SAT score rising from 1240 to 1360 ; the number of minority students more than doubled ; the endowment grew from $350 million to more than $3 billion ; the annual operating budget rose from $177 million to more than $650 million ; and annual research funding improved from $15 million to more than $70 million
danyaljj commented 6 years ago

These are some questions from the SQuAD dataset and are supposed to be one sentence but are split into two sentences. I understand that some of them not are "standard", but after all any "human" would consider them one sentence and our sentence-splitter should have the same behavior as well.

Who grouped 23 unpublished pieces and published them as Opp. 66-73 in 1855?
Who did a recording where Étude Op. 10 No. 5. follows relude Op. 28 No. 7?
The skit on JImmy Kimmel Live! was a depiction of Kanye West and what reporter?
How many credits were needed for an M.Sc. (Tech.) degree before the Bologna Process?
How many credits were needed for an M.Sc. from a traditional university prior to the Bologna Process?
When were millions of people supposedly exposed to e. coli and an anthrax-like bacteria?
where was producer mr. fingers' from?
when was mr. fingers' "mystery of love" released? 
when was mr. fingers' "can you feel it?" released?
george centeno, darren ramirez, and martin o. cairo developed a hard house sound in what city?
When did Public Safety Canada partner with Stop.Think. Connect?
Kiss' double live album Alive! came out in what year?
By the 1720s. how many cafes were in Paris?
What are sexual attraction. behavior, and idenity components of?
When did Compute! report the number of systems sold at 7 million?
What year was the single "Hey Hey! Holy Mackerel! (The Cubs Song)" produced?
Who is the author of "Who Governs? Democracy an Power in An American City"
Which new BYU school was added under Pres. Dallin H. Oaks in 1971?
What was believed to be one of the schools greatest strengths by Pres. Jeffrey R. Holland?
What did Pres. Bateman replace a scheduled devotial to do following the Sept. 11th, 2001 attacks? 
What did BYU's athletic program join under Pres. Bateman?
Who shouted "Eureka!" while checking the purity of a crown?
What is federalist paper no. 39?
What is the maximum amount of load a USB 1. and 2.0 device may draw?
Which CBC affiliate joined E! in February 2006?
About how thick would a PCB layer be if it contained three oz. of copper?
What happened during the c. 488–444 Ma and early Silurian period?
screen shot 2018-03-19 at 11 14 10 am
danyaljj commented 6 years ago

Another mistake: " A Science Hall was built in 1883 under the direction of Fr. Zahm, but in 1950 it was converted to a student union building and named LaFortune Center, after Joseph LaFortune, an oil executive from Tulsa, Oklahoma."

Splits on "Fr."