datatogether / sentry

Parallelized web crawler written in Golang

GNU Affero General Public License v3.0

14 stars 6 forks source link

Add crawler_test.go and some static examples. #50

Closed chinchila closed 6 years ago

chinchila commented 7 years ago

Changes

static pages with some links that should and shouldn't be crawled.
test cases for content and main crawlers.
setup & teardown database and configuration for testing.

Issue

38

dcwalk commented 6 years ago

Hey @b5 are you able to review when you get the chance?

b5 commented 6 years ago

Thanks so much for this PR @chinchila, I'd like to check with @dcwalk to make sure we're good, and unless there's an issue we'll get this merged. Thanks again!

dcwalk commented 6 years ago

This is great! Sorry for missing your notification earlier @b5 👍