Open pm5 opened 4 years ago
it was because majority of kknews.com and read01.com article would first 302 to a "loading page" before reaching the actual content page. So a lot of the htmls we collected are the loading page.
I see. Maybe we can look into that again when we have some time.
I recently found that you could actually read kknews.cc and read01.com articles with text-mode browsers such as w3m (or lynx, but not elinks, curiously), which means we probably don't need to run javascript to scrape their contents:
Was it because of proxies or some other problems that we need selenium to scrape these two sites in the first place?
toutiao.com remains javascript-only, so personally I think this issue is of a lower priority since we are not getting away with selenium all together anyway.