search

nietaki / crawlie

A simple Elixir library for writing decently-performing crawlers with minimum effort.
MIT License
89 stars 11 forks source link

Have crawlie operate in the library's supervision tree instead of the caller's #12

Closed nietaki closed 7 years ago

nietaki commented 7 years ago

There seems to be some work done by the GenStage guys for it: https://github.com/elixir-lang/gen_stage/blob/master/examples/consumer_supervisor.exs

nietaki commented 7 years ago

After some more consideration, I don't think it's a good idea.

The Flow returned by Crawlie is flexible to work with and can be used in any supervision structure the user sees fit. Flow also provides helpers for running flows as a separate process, instead of just "locally". Running it as part of Crawlie's supervision tree would limit the flexibility and I don't see much benefit to it anymore.