search

skrapeit / skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
https://docs.skrape.it
MIT License
805 stars 59 forks source link

[QUESTION] Unexpected encoded path? #141

Closed oxiadenine closed 3 years ago

oxiadenine commented 3 years ago

I'm trying to use scrapeit for the first time and it's throwing an error like:

Exception in thread "main" java.lang.IllegalArgumentException: unexpected encodedPath:

val extracted = skrape(HttpFetcher) {
    request {
        url = config.getString("url")
    }
    extract { 
        htmlDocument {
            h1 { findFirst { text } }
        } 
    }
}

Also, I don't see the option to ignoreContentType, since maybe it's the problem.

oxiadenine commented 3 years ago

Fixed, error is caused by not adding a '/' to the end of the Url

christian-draeger commented 3 years ago

Hey glad to hear you are giving skrape{it} a try :) This is a valid workaround for the HttpFetcher (problem should not exist for BrowserFetcher or AsyncFetcher).

It is already fixed on the master and will be part of the 1.1.0 release that will be published within the next days. Since version 1.1.0 the "/" workaround should not be needed anymore.