We now have a functionality that allows bulk inserts of documents. In other words, rather than returning one big document, scrapers can just yield a list of documents.
Non-RSS scrapers that scrape massive amounts of data and take a loooooong time to run (forums, reviews) should be updated to make use of this functionality. It should also be added as best practice to the doc/ folder.
We now have a functionality that allows bulk inserts of documents. In other words, rather than returning one big document, scrapers can just yield a list of documents. Non-RSS scrapers that scrape massive amounts of data and take a loooooong time to run (forums, reviews) should be updated to make use of this functionality. It should also be added as best practice to the
doc/folder.See this commit: https://github.com/uvacw/inca/commit/e3c7449fbd4425df3f864329d72a94833e7a1d51#diff-7d1847a2a717c07e47cf6be80d83b6bb