pySBD spaCy pipeline component uses a token-based approach and sets is_sent_start to True or False depending on Spans obtained from pySBD character offsets. We create Span objects using doc.char_span method by creating a slice - doc.text[start:end] which is a sentence span whose first Token object needs to have attribute is_sent_start set to True. On the other hand, if the character indices don’t map to a valid span it returns None . Hence we get irregularities in pySBD & pySBD + spaCy sentence output.
nipunsadvilkar
commented
4 years ago
pySBD spaCy pipeline component uses a token-based approach and sets
is_sent_starttoTrueorFalsedepending onSpans obtained from pySBD character offsets. We createSpanobjects usingdoc.char_spanmethod by creating a slice -doc.text[start:end]which is a sentence span whose firstTokenobject needs to have attributeis_sent_startset toTrue. On the other hand, if the character indices don’t map to a valid span it returnsNone. Hence we get irregularities in pySBD & pySBD + spaCy sentence output.The inability to get
Spanobject from pySBD character offsets can be tackled using the deconstruction ofDocobject like the way PKSHATechnology-Research/camphr authors have writtenget_doc_char_spanwhich usesdestruct_token