Need to put await - Githubissues

astelmach01 / GPT-crawler-backend-python

2 stars 1 forks source link

Need to put await #15

Open th-karanveer opened 2 months ago

th-karanveer commented 2 months ago

https://github.com/astelmach01/GPT-crawler-backend-python/blob/813d528b7c08b104011030acab392ae0866db4be/app/web/api/core/crawl.py#L90

hamza-cskn commented 2 weeks ago

I agree but If 200 links are found on a page, it will block too long. We need something like waitForAll() (to run parallelly) or MAXIMUM_LINK_COUNT_PER_PAGE. The depth setting is not enough.