BIDS-projects / scraper

Collects data from websites of data science institutions
2 stars 0 forks source link

introduce depth limit #17

Closed don-han closed 8 years ago

don-han commented 8 years ago

set the level of how deep a crawler goes

Assumption: Most interesting information exists at the nearest depth to the base url

Reasoning: As you go deeper into the website, the information becomes more specialized to specific function such as the integration service #2 such that the information no longer applies to the description of the institution

don-han commented 8 years ago

Now that scraper has both the tier limit and the page request limit, I don't think we need a third limitation.