A web app like luongthang.net_webapp.py should have only these responsibilities: processing authentication/authorization, creating routers for specific API endpoints, interacting with crawler's abstraction rather than specific crawler for specific sites.
Adding CI/CD
Cuz this is quite new, consider switching to another async web framework to gain benefit in concurrently crawling.
Some thoughts:
luongthang.net_webapp.pyshould have only these responsibilities: processing authentication/authorization, creating routers for specific API endpoints, interacting with crawler's abstraction rather than specific crawler for specific sites.