search

brackets-archive / brackets.io

brackets.io website
0 stars 0 forks source link

[CLOSED] Update robots.txt #98

Open core-ai-bot opened 3 years ago

core-ai-bot commented 3 years ago

Issue by ghost Sunday Mar 08, 2015 at 20:04 GMT Originally opened as https://github.com/adobe/brackets.io/pull/144

ghost included the following code: https://github.com/adobe/brackets.io/pull/144/commits

core-ai-bot commented 3 years ago

Comment by peterflynn Monday Mar 09, 2015 at 06:10 GMT

What is this change needed for? Does it have any effect? It seems like all the user-agent categories allow the same access.

core-ai-bot commented 3 years ago

Comment by ghost Monday Mar 09, 2015 at 06:21 GMT

From the distance it seems to have no effect. Specific mention allows those bots to delay their crawl, and it eases the host server. I had skipped the "Crawl-Delay: 20" earlier.

core-ai-bot commented 3 years ago

Comment by peterflynn Monday Mar 09, 2015 at 06:36 GMT

I don't think we've ever had performance problems due to server load form webcrawlers, so I'm not sure this change is necessary.

Also -- if we were to make a change like that -- I'd think you could just set Crawl-Delay globally instead of having to call out each individual crawler separately...

core-ai-bot commented 3 years ago

Comment by ghost Monday Mar 09, 2015 at 06:40 GMT

Made a global call instead, as suggested. When you can save few bits, why not save it? @peterflynn

core-ai-bot commented 3 years ago

Comment by peterflynn Monday Mar 09, 2015 at 07:28 GMT

I guess I'm not clear on how it saves bits, though -- it just throttles how quickly webcrawlers make requests, right? They'll still request the same number of pages (same amount of bits), just more slowly.

core-ai-bot commented 3 years ago

Comment by ghost Monday Mar 09, 2015 at 12:38 GMT

I had done my high school senior year project on Bandwidth & Crawlers. I had taken two VPS with same configurations and hosted, updated same amount of content but the one with crawl delay enable had lower bandwidth needs hence saving my bills. Bandwidth in other way are data transferred. If a huge website with minor change had to synced frequently you obviously have more data to transfer. While in this case, the change is very minor and does not have much significance but as I concluded in my study, a delayed crawler if the website is not updated frequently (read news portals) is a best practice.

core-ai-bot commented 3 years ago

Comment by ghost Tuesday Mar 10, 2015 at 06:06 GMT

Looking forward to your review call.

core-ai-bot commented 3 years ago

Comment by ficristo Friday Sep 23, 2016 at 18:13 GMT

Seems the user has deleted his account. Closing