[UN corpus] 下载联合国digital library的pdf并且转化成文本格式上传huggingface

[liyongsea]

https://digitallibrary.un.org/?ln=en 目前包含

• 约400G的pdf
• 共？？多余种的文档
• 20个6语言的官方文档

目标是下载所有的文档并转成文本格式，上传huggingface

Update 0413 - download data after 2000

• we realize some pdf might contain image, to simplify the task, we now focus on pdf before year 2000

method

• get all url from sitemap, crown the pdf date from the website (need to run ~12h. where is the code ?)
• crown pdf according to their date

related data

• 3 millions urls (html) available on https://digitallibrary.un.org/?ln=en
• 223K documents (correspond to 7x223K urls in the sitemap) with 6 official languages of UN (we call it parallel url)
• 146K after 2000 AC (could be converted easily into text)
• ?? analytics on the converted text

[liyongsea]

• wangran make sure the text convertion script is correctly working
• update the text data onto huggingface (??G)
• find vm to download the remaining data

[Wzixiao]

Update 0427

1. 根据时间排序完毕的链接获取dataset地址 "https://huggingface.co/datasets/ranWang/UN_PDF_RECORD_SET"
2. 2000年之后的url信息dataset地址同上
3. pdf文件全部下载完毕
4. 提取pdf文字全部完成
5. 分批上传dataset完成
6. 上传完成，最新dataset地址 “https://huggingface.co/datasets/ranWang/UN_Historical_PDF_Article_Text_Corpus”

[liyongsea]

pdf to text script is missing

[liyongsea]

• 做一个年代文件数的histogramme
• 看一下2000年以前的文档有没有能用的

[Wzixiao]

新的数据源:https://documents.un.org

new issue: https://github.com/liyongsea/parallel_corpus_mnbvc/issues/48