search

fredwu / crawler

A high performance web crawler / scraper in Elixir.
945 stars 91 forks source link

Why not use floki as a DSL for scraping page content ? #1

Closed merongivian closed 7 years ago

merongivian commented 7 years ago

Would u consider using it? i can contribute with this

sensiblearts commented 7 years ago

Floki is in the dependencies -- is being used by the Parser.

I am guessing that, by "DSL for scraping page content," Fred means a DSL at a higher level than Floki; e.g., what to aggregate, or save.

fredwu commented 7 years ago

Hi @merongivian,

@sensiblearts is right - Floki is being used by the parser at the moment. Once I'm ready to implement the scraping feature I will think about how to utilise Floki a bit more, in the form of a higher level DSL or API.

Hope that makes sense. :)

merongivian commented 7 years ago

got it, just read the codebase so its clear now :)