MichaelRoeder
commented
5 years ago - Restructured the ip- and domain-based queues. Introduced the general idea of URI groups and their handling within an abstract class.
- Fixed the deletion process of URI sets (i.e., they are deleted after a worker states that it crawled the IPs instead of being directly removed from the queue as soon as a worker requests them).
- Removed Rethink DB classes.
- Removed the usage of URI-Type from the MongoDB implementations.
- Fixed several smaller things in the queues.