Closed AbdulrhmnGhanem closed 10 months ago
A text normalizer should be run on the model output before doing any postprocessing steps.
def nomalize(text: str) -> str: ... example = "الذين كذبوا بالكتاب الذين كذبوا بالكتاب وبما أرسلنا به رسلنا فسوف يعلمونا" assert normalize(example) == "الذين كذبوا بالكتاب وبما أرسلنا به رسلنا فسوف يعلمونا"
You can test this function implementation here
A text normalizer should be run on the model output before doing any postprocessing steps.