Evaluation steps:

[ ] Evaluate the indexing component as well.
- [ ] Test numbers
- [ ] Test time needed to index
- [ ] Use different configurations and show the results and time.
[ ] Evaluate the UI Design by dogfooding it to my friends.
[ ] Mention computer configuration and process
- [ ] CPU Family Intel Pentium B960 Dual Core Intel Core Duo E8400
[ ] Look at the Distributed Web-Crawler evaluation section.
[ ] Look at the Results of a Large Crawl section in design and implementation of a high performance distributed web crawler.
[x] Consider using this website for testing: https://crawler-test.com/
[x] Consider testing coverage by testing random links and note of they are crawled or not?
[x] Choose 2 E-commerce websites and test them; prepare the expected result of each and discuss challenges.
[x] Jobs offers websites.
[x] Social media is an issue. Could you show it why and how to solve this?
[x] How long does it take to crawl the whole product?
[x] How long does it take to crawl the whole product with passing only the base url?
[ ] Use scrapy to compare the performance!
[ ] Test if the configurations are correct and the result is correct, for example, the ranking and the search result.
[ ] Discuss the result correctly.
[ ] Show how easy to add a container to reduce time to half!
[ ] Compare using simple CURL commands VS Selenium. Show why rendering can explore more information that CURL can not!

Metrices:

[ ] Number of visited URLS
[ ] Number of duplicated URLS.
[ ] Number of documents found (Products).
[ ] Number of Errors and list them.
[ ] Number of DB requests.
[ ] Time spent
[ ] CPU/RAM
[ ] Number of Nodes needed
[ ] The effect of adding an extra Node! Try having two different websites against making same nodes compete to crawl same website.