Update crawler configuration options.

[steveworley]

Expose a number of crawler configuration options so they can be controlled via the crawler configuration yaml file.

• Adds timeout
• Adds connect_timeout
• Adds verify
• Adds cookies