Closed ghost closed 8 years ago
From the distance it seems to have no effect. Specific mention allows those bots to delay their crawl, and it eases the host server. I had skipped the "Crawl-Delay: 20" earlier.
I don't think we've ever had performance problems due to server load form webcrawlers, so I'm not sure this change is necessary.
Also -- if we were to make a change like that -- I'd think you could just set Crawl-Delay
globally instead of having to call out each individual crawler separately...
Made a global call instead, as suggested. When you can save few bits, why not save it? @peterflynn
I guess I'm not clear on how it saves bits, though -- it just throttles how quickly webcrawlers make requests, right? They'll still request the same number of pages (same amount of bits), just more slowly.
I had done my high school senior year project on Bandwidth & Crawlers. I had taken two VPS with same configurations and hosted, updated same amount of content but the one with crawl delay enable had lower bandwidth needs hence saving my bills. Bandwidth in other way are data transferred. If a huge website with minor change had to synced frequently you obviously have more data to transfer. While in this case, the change is very minor and does not have much significance but as I concluded in my study, a delayed crawler if the website is not updated frequently (read news portals) is a best practice.
Looking forward to your review call.
Seems the user has deleted his account. Closing
What is this change needed for? Does it have any effect? It seems like all the user-agent categories allow the same access.