CyberspaceSpider is a visualization-based web crawling project that maps the path a web crawler takes as it navigates through the internet. With CyberspaceSpider, you can gain insights into the structure of the web and the relationships between different sites. It is a simple and intuitive tool that provides a unique perspective on web crawling.
https://botcheck.luminati.io/
https://brightdata.com/blog/how-tos/user-agents-for-web-scraping-101