Create dataset du_reader

[albertvillanova]

• uid: du_reader
• type: processed
• description:
  • name: DuReader
  • description: DuReader is a large-scale real-world Chinese dataset for Machine Reading Comprehension (MRC) and Question Answering (QA).
  • homepage: https://ai.baidu.com/broad/introduction?dataset=dureader
  • validated: True
• languages:
  • language_names:
  • Chinese
  • language_comments:
  • language_locations:
  • China
  • validated: False
• custodian:
  • name: Baidu.com
  • in_catalogue:
  • type: A commercial entity
  • location: China
  • contact_name: Jing Liu
  • contact_email: liujing46@baidu.com
  • contact_submitter: False
  • additional: https://ir.baidu.com/
  • validated: False
• availability:
  • procurement:
  • for_download: Yes - it has a direct download link or links
  • download_url: http://ai.baidu.com/broad/download?dataset=dureader
  • download_email:
  • licensing:
  • has_licenses: Yes
  • license_text: Copyright 2017 Baidu.com, Inc. All Rights Reserved Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
  • license_properties:
    • copyright - all rights reserved
  • license_list:
    • apache-2.0: Apache License 2.0
  • pii:
  • has_pii: Unclear
  • generic_pii_likely:
  • generic_pii_list:
  • numeric_pii_likely:
  • numeric_pii_list:
  • sensitive_pii_likely:
  • sensitive_pii_list:
  • no_pii_justification_class: general knowledge not written by or referring to private persons
  • no_pii_justification_text:
  • validated: False
• processed_from_primary:
  • from_primary: Original data
  • primary_availability:
  • primary_license:
  • primary_types:
  • validated: False
• media:
  • category:
  • text
  • text_format:
  • other
  • JSON
  • audiovisual_format:
  • image_format:
  • database_format:
  • .ZIP
  • text_is_transcribed: No
  • instance_type: A 4-tuple: a question, a question type, related documents, and a human annotated answer
  • instance_count: 100K<n<1M
  • instance_size: 100<n<10,000
  • validated: False
• fname: du_reader.json

[albertvillanova]

DONE: https://huggingface.co/datasets/bigscience-catalogue-data/du_reader_lm

Sample:

{'text': '茄子卤面的做法如下一、材料虾仁100g、茄子150g、面条350g、食盐适量、酱油适量、味精适量、姜适量、蚝油适量、调和油适量、柿子椒适量、洋葱适量、黑木耳适量、苦菜适量、香菇适量、胡萝卜适量、鸡蛋1只。二、做法1、虾去头去皮去肠，洗净备用。2、干香菇提前洗干净泡发，切成块继续放在水里泡。3、茄子去皮切成片。4、木耳、黄花菜提前泡发好回刀。洋葱、胡萝卜、辣椒切成块。5、炒锅放油加热，茄子入锅煎后盛出。6、炒锅放油加热，姜丝入锅煸香后，加虾仁、洋葱、茄子、辣椒、胡萝卜、木耳、黄花菜酱油、蚝油入锅翻炒。7、香菇连同泡香菇的水一同入锅。8、再加适量的水，盖上锅盖烧开，加盐、味精，搅拌均匀。9、淋入芡汁、鸡蛋，搅拌均匀，再煮开后关火。10、煮锅加水烧开，煮熟面条。11、面条出锅，配上卤、黄瓜即可。'}

[mariosasko]

self-assign

[mariosasko]

The LM script is available here: https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_zh_du_reader

[albertvillanova]

Thanks @mariosasko