search

hfthair / emerald_crawler

10 stars 2 forks source link

cannot download the dataset #6

Closed zhehengluoK closed 2 years ago

zhehengluoK commented 2 years ago

hi, thanks for offering the dataset. Though I can open the full paper via the link, I cannot find anything like ezproxy in my cookies, so only about 13 thousands papers are available for me to download. Is there any other way that could give me access to the whole dataset? Thanks.

memray commented 2 years ago

13k is a big bulk so I don't think it's the problem of cookie. Can you try locating the papers that you failed to download and check if you can see their full text in the browser? It might be the case that your university/institution doesn't buy access to the other papers. We also released a list of Open-access papers (https://github.com/hfthair/emerald_crawler/blob/main/data/test_oa.urls) and they should be fully accessible.

Rui