Closed gesof closed 4 years ago
--no-scoring
and --no-incoming
means what db worker just generates batches for spiders (spider feed).--no-scoring
and --no-batches
means what db worker reads spider log and saves metadata.--no-batches
and --no-incoming
means what db worker reads scoring log from strategy worker and updates queue.In the tutorial 2nd is not necessary and only one db worker is started, that's why only --no-incoming
flag specified. DB worker with such flag generates spider feed and processes scoring log.
Good. And why not updating the doc with this description? Instead, a warning gets thrown letting the users in confusion.
Sure, guys. PRs are always welcomed.
A.
22 июня 2020 г., в 17:35, Gesof notifications@github.com написал(а):
Good. And why not updating the doc with this description? Instead, a warning gets thrown letting the users in confusion.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
Added the options in a PR
I think it is an issue with the documentation. According to https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html#starting-the-cluster
# Optionally, start next one dedicated to spider log processing. $ python -m frontera.worker.db --no-batches --config [db worker config module]
That throws a: [db-worker] Can't identify DB worker type (no-scoring False, no-batches True, no-incoming False)
Looking into the source code:
if no_batches and no_scoring: db_worker_type = 'linksdb' elif no_batches and no_incoming: db_worker_type = 'scoring' elif no_incoming and no_scoring: db_worker_type = 'batchgen' else: logger.warning("Can't identify DB worker type " "(no-scoring {}, no-batches {}, no-incoming {})" .format(no_scoring, no_batches, no_incoming)) db_worker_type = 'none'
So I believe that beside --no-batches one should also provide --no-scoring or --no-incomming as per the code above. Still, I am not sure which or why.