felipecsl / wombat

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
https://felipecsl.github.io/wombat/
MIT License
1.31k stars 129 forks source link

Is it possible to programmatically create the properties to search for? #54

Closed Carpela closed 8 years ago

Carpela commented 8 years ago

I have a set of sites I index regularly where I store various field definitions alongside xpath parameters. e.g.

   {"field_name" => "title", "xpath" => "//div[@itemprop=\"title\"]"}
    {"field_name" => "description", "xpath" => "//div[contains(@class,'description')]"}

Is there a way I can use that with the wombat DSL? Can't work out how to make the field_name display itself in the right way.

Carpela commented 8 years ago

I think I've got it.

Assuming an array of those above

Wombat.crawl do 
    base_url @base_url
    field_definitions.each do |fd|
        send(fd["field_name"],xpath:fd["xpath"])
    end
end
jnicho02 commented 8 years ago

If that works for you then you should probably close the issue....