clulab / processors

Natural Language Processors
https://clulab.github.io/processors/
417 stars 101 forks source link

Quote is shoved to next sentence #784

Open kwalcock opened 5 months ago

kwalcock commented 5 months ago

With a sentence like

“The assets of our republic were bought at very low prices by intermediaries, including by him, and were resold at a much higher value.”

the last quote is moved to its own sentence, whether the quotes are "smart" or not.

Sentence #0:
Tokens: (",0) (The,1) (assets,2) (of,3) (our,4) (republic,5) (were,6) (bought,7) (at,8) (very,9) (low,10) (prices,11) (by,12) (intermediaries,13) (,,14) (including,15) (by,16) (him,17) (,,18) (and,19) (were,20) (resold,21) (at,22) (a,23) (much,24) (higher,25) (value,26) (.,27)

Sentence #1:
Tokens: (",0)

I'm seeing a fair number of sentences that are just single quotes, the trailing ones.