usc-isi-i2 / dig-etl-engine

Download DIG to run on your laptop or server.
http://usc-isi-i2.github.io/dig/
MIT License
101 stars 39 forks source link

Round-robin TLD when adding docs to the kafka input queue #96

Open GreatYYX opened 6 years ago

GreatYYX commented 6 years ago

Round robin is not efficient in our current design. Another problem is that when the daemon thread finds out desired num is greater than actual num, it will put all data to queue at one time. A better solution would be: add at most N docs every time.