[SIIT] Crawl alternative data

[ArthurMinovsky]

Crawl data

1. Collect any data that suitable for our LLM

• Check quality of data // Should collect or not with the amount, available for collection, quality of text; paragraph, content, etc.

• Check Overlap by collected data list

• [ ] Output: List of recommended data for crawl

2. Data format should be

• Huggingface format dataset >This is RFC task .You can propose the method that use for convert to Huggingface data set format.
• for example of format you can explore with code

`from datasets import load_dataset

thsum = load_dataset('thaisum')`

Output:

• Huggingface data set format

[ArthurMinovsky]

Example of data source :

• https://tna.mcot.net/
• https://www.mfa.go.th/