Test all crawlers - Githubissues

Description

Crawler tests have been separated from the original issue and are now posted here:

[ ] Unit test (e.g. checking a loop, method, or function is working as intended)
[x] Integration test (e.g. checking if a certain group or set of functionality is working as intended)
[ ] Regression test (e.g. checking if by adding or removing a module of code allows other systems to continue to function as intended)
[ ] Stress test (e.g. checking to see how well a system performs under various situations, including heavy usage)
[ ] Performance test (e.g. checking to see how efficient a system is as performing the intended task)
[ ] Other...

We expect all the above tests to pass.

None