matthewmueller / x-ray

The next web scraper. See through the <html> noise.
MIT License
5.87k stars 349 forks source link

nested paginate support #340

Closed haysclark closed 5 years ago

haysclark commented 5 years ago

Is it possible to have nested paginate() calls?

I am trying to use X-ray to crawl from one paginated source to a 2nd paginated source, but I am not having any luck getting it to work. Is this a known limitation?

const x = Xray()
x('some_url', "#table-designer > tbody > tr", [{
        id: "td",
        data: x("td > a@href", {
          title: "h1",
          photo: "#content > img@src",
          models: x("#content > div > div > div > table.table.table-striped > tbody > tr", [{
              name: "td > a",
              url: "td > a@href"
          }])
          // inner paginated content 
          .paginate("#content > div > div > ul.pagination > li:last-child > a@href")
        }),
      }
    ])
    // outter paginated content
    .paginate("#content > div > ul.pagination > li:last-child > a@href");
lathropd commented 5 years ago

can you send me a private gist of the nonworking code?

daniel.lathrop@gmail.com