matryer / vice

Go channels at horizontal scale (powered by message queues)
https://medium.com/@matryer/introducing-vice-go-channels-across-many-machines-bcac1147d7e2
Apache License 2.0
1.54k stars 79 forks source link

Question about processing file between multiple nodes #55

Open dertin opened 5 years ago

dertin commented 5 years ago

Hi, I have a huge file, on each line I have a value to send for a GET request.

I would like to create a queue with the file information, from a master server. and to be able to connect different nodes so that they take from the queue of messages, the values ​​to make the GET request and save the answer.

The nodes should not take the same message to process. double work should be avoided. If a node can not make the request, it could include the message in the queue again.

Can your library be useful to me? I am new programming in go lang. Can you give me some recommendation?

Regards