Refactor the "crawl.py" file to improve maintainability and organisation

[Am0stafa]

Description:

The crawl.py file in our project contains 31 functions and over 500 lines of code. While this file may have started out as a convenient place to put all of our crawling functions, it will become unwieldy and difficult to maintain as the project grow so it such be fixed as early as possible.

I propose refactoring crawl.py by splitting the functions into separate files based on their relation and functionality. Here are some benefits of doing so:

• Improved organization: Breaking up the code into smaller, more focused files will make it easier to find and modify specific functions. It will also make the codebase easier to navigate for new team members.

• Easier testing: Smaller files with focused functions make it easier to test individual pieces of functionality. This will improve our testing coverage and make it easier to catch bugs before they reach production.

• Reduced complexity: The size and complexity of crawl.py makes it difficult to reason about the code and understand how it all fits together. By breaking up the functions into separate files, we can reduce the cognitive load required to work with the codebase. Basically making the code more beginner-friendly

I propose that each file should contain a subset of related functions that perform a specific task. This will allow us to more easily reason about each file's purpose and the functions contained within it.

Related Issues

2

[HarshvMahawar]

I want to work on this issue. I think the main challenge to refactoring the crawl.py file would be understanding the dependencies and effective error handling of the functions in crawl.py to other code files. Are there any resources about the project structure or anything to understand the code base better? I would love to dive into those...thank you:)

[mshudrak]

Hi @HarshvMahawar, we have multiple PRs addressing this issue. I'd encourage you to consider other issues to work on.