search

ContentMine / quickscrape

A scraping command line tool for the modern web
MIT License
259 stars 42 forks source link

Fix url list processing #66

Open zemanel opened 8 years ago

zemanel commented 8 years ago

When specifying a list of urls to process, through the urllist file param, the last url is not processed.

I believe its a race condition, whereas finish() is called on a setTimeout() tick, without the last url having finished processing.

I removed an finish call as it will be called on the ensuing ticks.

blahah commented 8 years ago

Thanks for this, I will test and merge asap

robintw commented 8 years ago

I see this hasn't been merged, but has the fix been included some other way? I can't seem to reproduce the problem.

(If it has been fixed, can this PR be closed?)