Most settings are commented out in this file, the only one I personally enabled is the UserAgent.
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'OurCanadianCrawler (+http://wiki.ourcanadian.ca)'
We will need to go through each of these settings and determine how they apply to us and what are the best options for us to optimize our crawling. The file lists all the relevant documentation links.
Path
ocse-core/coast_to_coast/coast_to_coast/settings.pyMost settings are commented out in this file, the only one I personally enabled is the UserAgent.
We will need to go through each of these settings and determine how they apply to us and what are the best options for us to optimize our crawling. The file lists all the relevant documentation links.