search

red-data-tools / red-datasets

A RubyGem that provides common datasets
MIT License
30 stars 25 forks source link

Support livedoor news dataset #25

Closed kou closed 2 years ago

kou commented 5 years ago

https://www.rondhuit.com/download.html#ldcc

kou commented 2 years ago

Directory structure:

text
├── CHANGES.txt
├── README.txt
├── dokujo-tsushin
│   ├── LICENSE.txt
│   ├── dokujo-tsushin-4778030.txt
│   ├── ...
├── it-life-hack
│   ├── LICENSE.txt
│   ├── it-life-hack-6292880.txt
│   ├── ...
...
└── topic-news
    ├── LICENSE.txt
    ├── topic-news-5903225.txt
    ├── ...
    └── topic-news-6918105.txt

Format for each .txt:

${URL}\n
${TIMESTAMP}\n
${BODY}

For example, text/topic-news/topic-news-6918105.txt:

http://news.livedoor.com/article/detail/6918105/
2012-09-04T12:45:00+0900
ジャンプ連載漫画が終了に、ユーザが新たな提案!?
『ONE PIECE』や『NARUTO』、『BLEACH』といった人気漫画でもお馴染み、...
kou commented 2 years ago

@abcdefg-1234567 This may be a good candidate for the next issue.