search

nipunsadvilkar / pySBD

🐍💯pySBD (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box.
MIT License
802 stars 83 forks source link

`--` breaks segmentation #125

Open micycle1 opened 9 months ago

micycle1 commented 9 months ago

Describe the bug Instances of -- often break the segmentation. When these are replaced, segmenting the same sentence works as expected.

To Reproduce

Breaking Examples:

1.

"Volumes within all major lending categories continued to grow, except credit card volumes, which leveled off -- a typical seasonal trend prior to the significant growth expected through the holiday season. Commercial and industrial (C&I) loans and commercial real estate (CRE) loans grew at a slower pace than during the same period one year ago but were two of the fastest growing loan categories recently, along with other consumer lending. Auto loans, mortgages, and home equity loans were up slightly during the period, but volumes in the latter two categories remained slightly below last year's volumes. On balance, banking contacts continued to report healthy loan portfolios and improving customer credit quality, although a few are starting to see scattered problem loans emerge. Several contacts expressed concerns about a potential overheated housing market, primarily for multifamily development in larger urban areas. In addition, a few banks have tightened credit standards since last period -- primarily for C&I lending and CRE."

2.

Most respondents noted capacity constraints in labor-intensive oilfield services where people have to be especially well trained and qualified--such as fracturing, pressure pumping or tubular makeup. In addition, rental rates have improved for rigs capable of drilling deep water because many rigs were moved to other areas of the world that promised greater returns. The oilfield services that could not leave with the rigs--such as downhole fluids, supply boats, and helicopter services--are in excess supply.

Output for 2 (is not segmented):

['Most respondents noted capacity constraints in labor-intensive oilfield services where people have to be especially well trained and qualified--such as fracturing, pressure pumping or tubular makeup. In addition, rental rates have improved for rigs capable of drilling deep water because many rigs were moved to other areas of the world that promised greater returns. The oilfield services that could not leave with the rigs--such as downhole fluids, supply boats, and helicopter services--are in excess supply.']

2b (-- replaced with —), this works as expected.

'Most respondents noted capacity constraints in labor-intensive oilfield services where people have to be especially well trained and qualified—such as fracturing, pressure pumping or tubular makeup. In addition, rental rates have improved for rigs capable of drilling deep water because many rigs were moved to other areas of the world that promised greater returns. The oilfield services that could not leave with the rigs—such as downhole fluids, supply boats, and helicopter services—are in excess supply.'

Expected output for 2b (is now segmented):

['Most respondents noted capacity constraints in labor-intensive oilfield services where people have to be especially well trained and qualified—such as fracturing, pressure pumping or tubular makeup.',
 'In addition, rental rates have improved for rigs capable of drilling deep water because many rigs were moved to other areas of the world that promised greater returns.',
 'The oilfield services that could not leave with the rigs—such as downhole fluids, supply boats, and helicopter services—are in excess supply.']