Scrapping + crawling is not synchronus

matthewmueller / x-ray

The next web scraper. See through the <html> noise.

MIT License

5.87k stars 349 forks source link

Scrapping + crawling is not synchronus #306

Closed AutomataVM closed 5 years ago

AutomataVM commented 6 years ago

Subject of the issue

When I scrape a collection of results, I want it to start crawling to the next page after it finishes scraping all the current data. However, I find that when it executes, crawling happens before showing scrapped data

Your environment

version of node: v6.11.0 version of npm: v3.10.10

Steps to reproduce

function scrapeAndCrawl(searchUrl) {

if (!searchUrl){
    formData.splice(0,1);
    startSearch();

lathropd commented 5 years ago

Calling .concurrency(1) should do this. See https://github.com/matthewmueller/x-ray/blob/master/Readme.md#xrayconcurrencyn