[Super AI] Make pdpa blind data code

[ArthurMinovsky]

Before training data, we want to blind people's information especially

1. Person Name
2. Personal Identification Number (บัตรประชาชน)

Input:

• Huggingface Datasets like https://huggingface.co/datasets/EleutherAI/pile

Output:

• Same Huggingface Dataset but with blinded text <person> <person_id>

Our code should work with the Huggingface dataset .map function to be easily parallelized on the Lanta node https://huggingface.co/docs/datasets/process

Our potential code candidates

• @i17bkk Code
• P Conan Twitter code
• P Chet code
• POS tagging model

Side Note: Example how to download oscar dataset from huggingface:

from datasets import load_dataset

def download():
    text_dataset = load_dataset("oscar", "unshuffled_deduplicated_th")

Please make sure you set export HF_DATASETS_CACHE="/folder_for_storing_data"