Previously the sentence was not able to tokenize the sentence, if the sentence ends with numeric character which i guess was a logic issue for checking and not splitting for decimal number.
I have changed logic a bit, specifically for not tokenizing in that condition.
So for the sentence - "India was declared a nation with its own constitution on 26 January 1950, while India gained independence on 14 August 1947. About 3 years went through the formation of the nation and the complete departure of the British." it's working fine also tested a few edge cases too.
Previously the sentence was not able to tokenize the sentence, if the sentence ends with numeric character which i guess was a logic issue for checking and not splitting for decimal number. I have changed logic a bit, specifically for not tokenizing in that condition. So for the sentence - "India was declared a nation with its own constitution on 26 January 1950, while India gained independence on 14 August 1947. About 3 years went through the formation of the nation and the complete departure of the British." it's working fine also tested a few edge cases too.