Preprocess text - Githubissues

Rocco000 / DarkModalGAN

0 stars 0 forks source link

Preprocess text #1

Open Rocco000 opened 2 days ago

Rocco000 commented 2 days ago

Preprocess textual data following these steps:

Transform Text:

[ ] Convert all text to lowercase for uniformity.
[ ] Translate non-English text into English.

Replace Patterns: Define and implement functions for domain-specific replacements

[ ] Expand common acronyms (aka, no. 2) into their full form.
[ ] Expand domain-specific acronyms (e.g. CBD -> cannabidiol)
[ ] Replace concentration annotation (e.g. 2ml/10gr ->2 ml per 10 gr)
[ ] Replace quantity annotation (e.g. 10x -> 10 pieces)
[ ] Replace percentage annotation (e.g. 10% -> 10 percent)
[ ] Expand unit measurement acronyms (e.g. 10mg -> 10 micrograms)

Remove redundant/noisy information: Define and implement functions to clean the text by removing:

[ ] Prices and monetary values.
[ ] URLs or web links.
[ ] Unprintable or non-ASCII characters.
[ ] Expiration dates.
[ ] Promotional terms (such as deal, offer, promo, discount).
[ ] Vendor name.
[ ] Shipping information.
[ ] Macro-category label.
[ ] Special characters (such as !, ?, $)

For pre-trained model compatibility

[ ] Implement lemmatization to reduce words to their base or dictionary forms.
[ ] Tokenize the text using the tokenizer of the pre-trained model.
[ ] Summarize text that exceeds the maximum token length supported by the pre-trained model.
[ ] Expand or pad text with fewer tokens to reach a reasonable length for fair comparison.

dagshub[bot] commented 2 days ago

Join the discussion on DagsHub!