indrajithi / tiny-web-crawler

A simple and easy to use web crawler for Python
MIT License
55 stars 11 forks source link

What is the `main` function for? #27

Closed Mews closed 1 week ago

Mews commented 1 week ago

At the bottom of crawler.py, there is this piece of code:

def main() -> None:
    root_url = 'https://pypi.org/'
    max_links = 5

    crawler = Spider(root_url, max_links, save_to_file='out.json')
    print(Fore.GREEN + f"Crawling: {root_url}")
    crawler.start()

if __name__ == '__main__':
    main()

I'm just curious as to what the purpose of this being here is. Looking at it seems like a small piece of code to test the module, but if this is the case it should probably be on a separate file like examples.py, not on crawler.py (if it is meant to be in the source at all)

indrajithi commented 1 week ago

This was part of the original script when it was not a python package. I will remove it as part of refactoring #17

Mews commented 1 week ago

Okay makes sense