DeepCrawl Exploration and Setup

[juanmriosv]

https://conductor.deepcrawl.com

To review:

• [x] Video: Project & Crawl Setup
• [x] Video: Reporting Overview
• [x] TEXT: The Robots.txt Overwrite Feature: Use Case
• [x] How to Crawl AJAX Escaped Fragment Websites
• [x] Using Crawl Limits to Reduce Credit Usage
• [x] Using Custom Extraction
• [x] Crawling Multiple Domains Or Subdomains
• [x] Comparing A Test Website To A Live Website
• [x] How to Fix your Failed Website Crawls
• [x] Restricting a Crawl to Certain Pages
• [x] Modifying URLs And Stripping Parameters
• [x] How to Crawl a Website with DeepCrawl
• [x] Four Awesome Things You Can Do With Regex
• [x] Limiting the Size and Depth of a Crawl
• [x] Crawl A Mobile Website With DeepCrawl

[juanmriosv]

Adding to questions for conductor

[juanmriosv]

@c21hornak Done reviewing this documentation. All are a general introduction to all what DeepCrawl is capable of doing. I have mentioned Ken that I am done reviewing it, he will go ahead with the next step which is setting up a call with Conductor to go over DeepCrawl. This will probably happen on Thursday.

[juanmriosv]

First Crawl has been setup after call with Conductor (08.14.17) It will be a full crawl at a rate of 3 URL per second. With a static IP address:

This crawl was "site only" (did not include any integration)

[juanmriosv]

Setting up Second crawl with google analytics, Search Console, and sitemap integrations. Possible date: Thursday 22nd