shashiongithub
commented
5 years ago hi, We are not sure about the rights for releasing the whole BBC dataset. We are looking into that. For the moment, I have added that exception to the script. To avoid too many redirects, try using "timestamp_exactness".
"[Sep 19, 2018]: Each URL includes a timestamp (14 letters; year, month, date, hour, minute and second, e.g., 20130206080312) showing when it was stored. By default, it is set to 14 letters. You can lower this to improve your chances to download a BBC article. For example, with timestamp_exactness=4 the time stamp will be chopped to first 4 letters (i.e., 2013) and it will try to download a version of the article stored in 2013 (any month, date, hour, minute or second). Every time you rerun the download script, lower the timestamp_exactness (>= 1) to improve the chances of getting missed URLs from the previous step. However, lower the timestamp_exactness value, more time it will take to retrieve the article."
jiacheng-xu
marythomaa98
~/XSum/XSum-Dataset$ python scripts/download-bbc-articles.py Creating the download directory. Downloading URLs from the XSum-WebArxiveUrls.txt file: urls left to download: 226711 5.19% [11765/226711]Traceback (most recent call last): File "scripts/download-bbc-articles.py", line 253, in
main()
File "scripts/download-bbc-articles.py", line 248, in main
DownloadMode(urls_file_to_download, missing_urls_file, downloads_dir, args.request_parallelism, args.timestamp_exactness)
File "scripts/download-bbc-articles.py", line 198, in DownloadMode
for url, story_html in results:
File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.
I know probably I can fix this issue but I want to make sure the copy of the dataset is exactly the same as you use in the paper. Could you just upload the dataset to Dropbox or Google cloud? or fix this issue. thx.