search

felipecsl / wombat

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
https://felipecsl.github.io/wombat/
MIT License
1.31k stars 129 forks source link

Dynamically set base_url and path. #70

Closed imnithin closed 7 years ago

imnithin commented 7 years ago

This is more of a question,

Would it be possible for to set base_url and path dynamically/run time?

felipecsl commented 7 years ago

No, this is not currently possible, sorry.

khacluan commented 4 years ago

you can do it

     def crawl(path)
        raw_code = <<-CODE
           base_url "https://www.github.com"
           path "#{path}"

           headline xpath: "//h1"
           subheading css: "p.alt-lead"

           what_is({ css: ".one-fourth h4" }, :list)
        CODE

        block = ->(c) { eval(raw_code) }

        Wombat.crawl &block
     end