Generalize crawler consumer

usc-isi-i2 / dig-etl-engine

Download DIG to run on your laptop or server.

http://usc-isi-i2.github.io/dig/

MIT License

101 stars 39 forks source link

Open GreatYYX opened 6 years ago

GreatYYX commented 6 years ago

only one general consumer is needed (no need to differentiate ACHE and RSS crawler)
user can specify which field name to be treated as project_name
user can define how many partitions and consumers for consumer