tefreeman
commented
6 years ago - Find an open source web crawler and modify it to fit our needs
- Store information into a database
- Needs to have mechanism to check if information has already been stored
- Needs to be able to work on multiple data sources
- Needs to allow us to target certain domains and page systems
- Needs to be multi-threaded so we can crawl at a decent speed