search

cavaliergopher / grab

A download manager package for Go
BSD 3-Clause "New" or "Revised" License
1.38k stars 151 forks source link

[Feature Request] client.DoChannel() #12

Closed oliverpool closed 7 years ago

oliverpool commented 7 years ago

Hi,

I'm considering using your library with a web crawler. One issue is that I don't know how many requests I will have when I start.

It could be great to have a client.DoChannel method (like DoBatch) which will consume a Request channel (until closing). This new method could be used to refactor DoBatch (which internally already uses a channel).

If you think that this could be a good idea, I would be happy to propose a PR.

cavaliercoder commented 7 years ago

I think this is a good idea. I think DoBatch started out this way but pivoted to the current slice-based invocation. I can't remember what the challenges were, but maybe your fresh perspective will find solutions I could not.

My only caution is that I haven't seen any precedent for other package APIs or standard libraries subscribing to channels. Typically channels are used inside the black box of a library and the interfaces use other primitives (in my observations so far). Do you know of any examples where this is not the case?

If you're happy to experiment, I'm happy to take a look at a PR. For now though, let's leave the implementation of DoBatch as-is and create DoChannel. We can refactor later if the Channel approach works well.

cavaliercoder commented 7 years ago

If you're building a web crawler, you may also be interested in https://github.com/cavaliercoder/spodermen. This project is a little dirty right now, but is being used in production at work and therefore getting lots of rapid development.

oliverpool commented 7 years ago

I can't remember what the challenges were, but maybe your fresh perspective will find solutions I could not.

The main problem I see, is that it is now the callers responsibility to close the channel.

My only caution is that I haven't seen any precedent for other package APIs or standard libraries subscribing to channels.

I don't know much about this, but I just found this (3-year old) article: https://inconshreveable.com/07-08-2014/principles-of-designing-go-apis-with-channels/

If you're building a web crawler

Thanks for the link! My goal is to download the new links when they appear (hence the need for your grab library, which support download resuming)