search

rflechner / ScrapySharp

reborn of https://bitbucket.org/rflechner/scrapysharp
MIT License
346 stars 75 forks source link

Unable to Scrape Dynamic Content #28

Closed TheBobFather1980 closed 4 years ago

TheBobFather1980 commented 5 years ago

I'm trying to use the ScrapingBrowser class to navigate and scrape a site. I noticed I was unable to enter the username and password on the login page because it is dynamically generated from Javascript. Is there a way to execute scripts on the page so I can get the full web page content?

coolhome commented 4 years ago

@TheBobFather1980 This library only works with server side rendered content (static) because HtmlAgilityPack is just an html parser. There is no real browser. You might want to look at something like Web Driver which uses a real browser. It should be noted that this approach has a lot more overhead costs.