[QUESTION] adding body in request

[MartinBirn]

After reading the documentation I did not find any mention of how to add a body to the request. Pardon me if I missed it.

[christian-draeger]

Hey, This is not supported by now since fetching websites rarely needs an request body, but I see the the point, especially when testing controllers that return html based on request body input.

I will add the feature as soon as possible

[christian-draeger]

Work in progress. i started to add body to request, the dsl will give you multiple possibilities to add a request body:

1 plain as string

most low level option, needs to set content-type header "by hand"

skrape(HttpFetcher) {
    request {
        url = "https://www.my-fancy.url"
        method = Method.GET
        headers = mapOf("Content-Type" to "application/json")
        body = """{"foo":"bar"}"""
    }
    extract {
        htmlDocument {
            ...

2 plain text with auto added content-type header that can be optionally overwritten

skrape(HttpFetcher) {
    request {
        url = "https://www.my-fancy.url"
        method = Method.POST
        body {
            data = "just a plain text" // content-type header will automatically set to "text/plain"
            contentType = "your-custom/content" // can optionally override content-type
        }
    }
    extract {
        htmlDocument {
            ...

3 with helper functions for json or xml bodies

supports json and xml autocompletion when using intellij 😎

skrape(HttpFetcher) {
    request {
        url = "https://www.my-fancy.url"
        method = Method.POST
        body {
            json("""{"foo":"bar"}""") // will automatically set content-type header to "application/json" 
            // or
            xml("<foo>bar</foo>") // will automatically set content-type header to "text/xml" 
            // or
            form("foo=bar") // will automatically set content-type header to "application/x-www-form-urlencoded" 
        }
    }
    extract {
        htmlDocument {
            ...

4 with on the fly created json via dsl 😁

skrape(HttpFetcher) {
    request {
        url = "https://www.my-fancy.url"
        method = Method.POST
        body {
            // will automatically set content-type header to "application/json"
            // will create {"foo":"bar","xxx":{"a":"b","c":[1,"d"]}} as request body
            json {
                "foo" to "bar"
                "xxx" to json {
                    "a" to "b"
                    "c" to listOf(1, "d")
                }
            }
        }
    }
    extract {
        htmlDocument {
            ...

5 with on the fly created form via dsl

skrape(HttpFetcher) {
    request {
        url = "https://www.my-fancy.url"
        method = Method.POST
        body {
            // will automatically set content-type header to "application/x-www-form-urlencoded"
            // will create foo=bar&xxx=1.5 as request body
            form {
                "foo" to "bar"
                "xxx" to 1.5
            }
        }
    }
    extract {
        htmlDocument {
            ...

To be done: Fetchers need to handle new request option

[christian-draeger]

done

will be part of next release