We'd like to lay down some infrastructure to allow us to configure the scraper in once place without modifying code. This will be less fragile and will prevent accidental commits when changing these properties.
Tasks
[ ] There will be two files in the repo root:
[ ] scraper.defaults.json
[ ] This file provides the default values for any config setting.
[ ] Each property in the file should have a ${PROPERTYNAME}.description property before it explaining it (what it does and some examples.) These will be ignored when parsing either config file.
[ ] scraper.config.json
[ ] This file provides the local-only overrides for the scraper config
[ ] When a property is not defined in this file, it should fallback to the defaults file.
[ ] This file should be ignored in .gitignore so it doesn't get commited by accident.
[ ] Initial config properties to support:
[ ] outputDirectory - where to output scraper files
[ ] useCache - whether to use cached HTML files when running the scraper
[ ] [optional] any others you feel are worth breaking out into this file.
Summary
We'd like to lay down some infrastructure to allow us to configure the scraper in once place without modifying code. This will be less fragile and will prevent accidental commits when changing these properties.
Tasks
scraper.defaults.json${PROPERTYNAME}.descriptionproperty before it explaining it (what it does and some examples.) These will be ignored when parsing either config file.scraper.config.json.gitignoreso it doesn't get commited by accident.outputDirectory- where to output scraper filesuseCache- whether to use cached HTML files when running the scraper