search

skrapeit / skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
https://docs.skrape.it
MIT License
805 stars 59 forks source link

DOM mode cannot be set #179

Closed pvegh closed 2 years ago

pvegh commented 2 years ago

This page describes how to request the page as DOM to be able to parse a JS rendered page https://docs.skrape.it/docs/dsl/extract-client-side-rendered-data

The example code doesn't work, mode cannot be set. Maybe the docs are outdated.

christian-draeger commented 2 years ago

Hey thx for pointing this out. The example code on the docs page is unfortunately outdated.

All documentation in the github projects readme is up-to-date though. I will try to update the docs page as soon as possible.

please have a look here until docs page is updated: https://github.com/skrapeit/skrape.it#scrape-a-client-side-rendered-page

pvegh commented 2 years ago

Thanks, looks like BrowserFetcher is all that's needed currently. I already tried it, it doesn't contain all the content that is generated by javascript at runtime on the page I need to parse.