Metadata is saved in distributed mode if there is a db worker with no flag --no-incoming. When I switched to single process mode, metadata is not saved. I did not find any setting that enables it. Is it indented? According to documentation metadata should be saved.
Here you can find my project. I am using discovery strategy. To add seeds, run python -m frontera.utils.add_seeds --config config.single --seeds-file seeds.txt. To start crawl, run scrapy crawl spider.
Metadata is saved in distributed mode if there is a db worker with no flag
--no-incoming
. When I switched to single process mode, metadata is not saved. I did not find any setting that enables it. Is it indented? According to documentation metadata should be saved.Here you can find my project. I am using discovery strategy. To add seeds, run
python -m frontera.utils.add_seeds --config config.single --seeds-file seeds.txt
. To start crawl, runscrapy crawl spider
.