remove PUNCT and COMMA token

ulf1 / augtxt

yet another text augmentation python package

Apache License 2.0

2 stars 0 forks source link

remove PUNCT and COMMA token #47

Closed ulf1 closed 2 years ago

ulf1 commented 2 years ago

sentence = 'Die Schülerin [MASK] ein Aufsatz, der sehr [MASK] war.'
augmented = re.sub('\s+', ' ', re.sub('[.?!;:,]+', ' ', sentence)).strip() 
augmented