I plan on doing a pass over all the crawlers and making them more homogenous where possible.
This should mostly be inconsequential stuff like the wording in log messages, but will also fix other issues like certain crawlers having wait times in the wrong place, or claiming that they've finished crawling even though you're actually just resetting the database, and no crawling is being done.
Additionally, I plan on moving functions around so they have a consistent ordering between crawlers, which should make the code a bit easier to read.
EDIT:
Alright so i don't lose track of where I am with this:
I plan on doing a pass over all the crawlers and making them more homogenous where possible.
This should mostly be inconsequential stuff like the wording in log messages, but will also fix other issues like certain crawlers having wait times in the wrong place, or claiming that they've finished crawling even though you're actually just resetting the database, and no crawling is being done.
Additionally, I plan on moving functions around so they have a consistent ordering between crawlers, which should make the code a bit easier to read.
EDIT: Alright so i don't lose track of where I am with this: