Housekeeping: Refactor the code base to a more Modular and Extensible Architecture

[indrajithi]

1. Separation of Concerns:

   • Separate the core crawling logic from the utilities and configurations.
   • Define interfaces or abstract classes for components that may need to be extended or swapped out.

2. Use of Dependency Injection:

   • Allow different components to be injected into the main crawler class, making it easy to extend or replace specific parts of the crawler.

3. Organize Code into Modules:

   • Organize the code into different files or modules based on their functionality.

4. Configure git hook to ensure all the test passes before push #20

5. Update how we import the package and update the readme usage doc

   • Make imports like from tiny_web_crawler import Spider Moved to #26

PRS:

34