Closed caccialdo closed 9 years ago
Chaining all the GET requests in the same Osmosis instance forces them to happen in that order one at a time. If you're giving Osmosis a bunch of URLs, then request each page with a separate Osmosis instance.
osmosis
.get('http://...').set({...}).data(fn)
osmosis
.get('http://...').set({...}).data(fn)
...
osmosis
.get('http://...').set({...}).data(fn)
Please use the latest changes for best results. A stack/queue incrementing issue has been fixed. The issue caused separate Osmosis instances to unnecessarily increase the request queue length.
Thanks for the swift answer. I used your recommended pattern instead, with the latest master. Worked flawlessly.
I wrote a script that generates a long command chain like this one:
I would have expected osmosis to kick off many concurrent connections (5, by default) but when I look at the
requests
property of theParser
instance when I run it, the number never gets past 1 and scraping seems to happen one url at a time indeed given the slowness.Am I missing something here or is it a genuine issue with this library?