skrapeit / skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
https://docs.skrape.it
MIT License
789 stars 57 forks source link

Allow suspend functions in DSL #214

Closed linean closed 1 year ago

linean commented 1 year ago

I've encountered a case where it would be helpful to use a suspend function directly from SkrapeItDsl.

Here is a simplified example:

skrape(AsyncFetcher) {
    request {
        method = Method.POST
        url = "https://www.someendpoint.com"
        headers = mapOf(
            "test" to suspendGetHeader()
        )
    }

    response {
        val testHeader = headers["test"]
        suspendSaveHeader(testHeader)
    }
}

The above code won't compile because suspend functions cannot be called from the DSL. Looking at the source code I've noticed this limitation is caused by the missing suspend modifiers and could be easily changed.

It's easy to solve this problem by extracting suspend functions outside of the DSL and everything will work, but I thought I'll propose adding support to the DSL. Please tell me what you think.

codecov[bot] commented 1 year ago

Codecov Report

Merging #214 (2f53f97) into master (3958869) will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #214   +/-   ##
=======================================
  Coverage   80.91%   80.91%           
=======================================
  Files          39       39           
  Lines        1168     1168           
  Branches      180      180           
=======================================
  Hits          945      945           
  Misses        162      162           
  Partials       61       61           
Impacted Files Coverage Δ
...tcher/src/main/kotlin/it/skrape/fetcher/Scraper.kt 80.96% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

christian-draeger commented 1 year ago

Thx for pointing this out :)