search

adobe / brackets.io

brackets.io website
111 stars 80 forks source link

Update robots.txt #144

Closed ghost closed 8 years ago

peterflynn commented 9 years ago

What is this change needed for? Does it have any effect? It seems like all the user-agent categories allow the same access.

ghost commented 9 years ago

From the distance it seems to have no effect. Specific mention allows those bots to delay their crawl, and it eases the host server. I had skipped the "Crawl-Delay: 20" earlier.

peterflynn commented 9 years ago

I don't think we've ever had performance problems due to server load form webcrawlers, so I'm not sure this change is necessary.

Also -- if we were to make a change like that -- I'd think you could just set Crawl-Delay globally instead of having to call out each individual crawler separately...

ghost commented 9 years ago

Made a global call instead, as suggested. When you can save few bits, why not save it? @peterflynn

peterflynn commented 9 years ago

I guess I'm not clear on how it saves bits, though -- it just throttles how quickly webcrawlers make requests, right? They'll still request the same number of pages (same amount of bits), just more slowly.

ghost commented 9 years ago

I had done my high school senior year project on Bandwidth & Crawlers. I had taken two VPS with same configurations and hosted, updated same amount of content but the one with crawl delay enable had lower bandwidth needs hence saving my bills. Bandwidth in other way are data transferred. If a huge website with minor change had to synced frequently you obviously have more data to transfer. While in this case, the change is very minor and does not have much significance but as I concluded in my study, a delayed crawler if the website is not updated frequently (read news portals) is a best practice.

ghost commented 9 years ago

Looking forward to your review call.

ficristo commented 8 years ago

Seems the user has deleted his account. Closing