Find a way to integrate a robots.txt handling in the core

[Psycojoker]

It would be a cool feature to be able to define robots.txt files:

• globally for the whole installation
• at the domain level
• at the application level

That would also mean: being able to integrate with applications that needs to add things to the robots.txt (maybe via a hook or a snippet system).