search

Steveorevo / node-red-contrib-nbrowser

Provides a virtual web browser (a.k.a. "headless browser") appearing as a node.
34 stars 13 forks source link

Scraping hangs #20

Closed KeiyaD closed 5 years ago

KeiyaD commented 5 years ago

I have an issue with scraping from a website HTML + values updated with Javascript. The website first shows a loading screen and after 5 seconds of loading, it shows the tables with measurements. The problem I am facing is that the scraping timeouts because it can't find the element with the option "Show browser window instance?" unchecked. With the function checked it works perfectly.

This is the element I am waiting to be visible: #page-content > section > div > div > div > div > div > div:nth-child(1) > div > table > tbody > tr:nth-child(10) > td:nth-child(2)

Any suggestions?

Steveorevo commented 5 years ago

This may not be an issue with nbrowser but the site itself; they may have custom javascript that won't update or needs to calc. the browser window size. Perhaps that's not possible when invisible. Try navigating to a blank page, set the browser window size, and then navigate to the given page. Likewise, you can delay prior to scraping.

Steveorevo commented 5 years ago

Marked invalid as we do not have enough steps/data to reproduce the issue.