Closed rossmounce closed 9 years ago
articles list here: https://github.com/rossmounce/quickscrapeijsem/blob/master/ijsemarticles.txt
It's interesting the log file only goes up to 64/1775 yet in the output folder directory it appears it got one or two further down the file list: subfolders for 64/1782 & 64/1802 are present
Problem definitely arose at 64/Pt_5/1775 : the output folder is empty except for a damaged/incomplete 83kb PDF file 56978.pdf (no .json files or anything else)
It seems there is no particular problem with scraping that URL - it worked fine for me. So there's a less well defined problem where quickscrape can hang. This is most likely something to do with handling network issues, which we could do better at.
I've made issues to track that (see above), so will close this one.
Previous issue arose from trying to diagnose this problem (the --url / -url thing wasn't the problem here). Here my bash:
It basically hung, without crash / exit on the 1000ish line of the ijsemarticles url list. I'll upload the list and link to it here in a bit.
I used Ctrl-C to stop the process. Would be good to print the current time to screen in one of the loglevel settings so I know when it hung. I have no idea when it stopped chugging away