Enhancement: autogenerate robots.txt

[Nightbr]

Hey, I have look at the code of SonataSeoBundle and I have not found any helper to generate the robots.txt.

Is it a planned feature or not ?

I handle it with a simple twig template so far:

# www.robotstxt.org/
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

User-agent: *
Disallow: /admin
Disallow: /sonatadmin
Disallow: /api
Sitemap: {{ url('sitemap_xml', {'_format':'xml'}) }}

But I think it could be a great feature. If it is planned to this bundle and nobody is on it, I can develop it and create a PR.

Anyone interested in this feature ?

[greg0ire]

I'm not aware of any plans for this bundle besides maintaining it, but any features are welcome.

[Nightbr]

Ok, I implement it in my current project and I will make a PR for the robots.txt features.

keep in touch on this ;)

[greg0ire]

Great!

[Nightbr]

Hey, here is my thought about robots.txt:

Specification sources:

• https://en.wikipedia.org/wiki/Robots_exclusion_standard
• http://www.robotstxt.org/norobots-rfc.txt

First, you can configure the robots.txt into the config.yml:

sonata_seo:
    robotstxt:
        -
            user-agent: [*]
            crawl-delay: 10
            disallow: /admin
            disallow: /sonatadmin
            allow: /api/doc
            disallow: /api
            sitemap: [https://www.example.com/sitemap_index.xml]
            host: www.example.com
        -
            user-agent: [BadRobot, GoogleBot]
            disallow: /private

For example, this configuration will generate this robots.txt:

# www.robotstxt.org/
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

User-agent: *
Crawl-delay: 10
Disallow: /admin
Disallow: /sonatadmin
Allow: /api/doc
Disallow: /api
Sitemap: https://www.example.com/sitemap_index.xml
Host www.example.com

User-agent: BadRobot
User-agent: GoogleBot
Disallow: /private

With this structure, I think we can cover all combination for a robots.txt file.

In the DependencyInjection/SonataSeoExtension.php, I will create a function configureRobotstxt(array $config, ContainerBuilder $container) in order to export the configuration in php array.

After this, I will create a folder Robotstxt, with a php file RobotstxtManager.php which will generate a valid robots.txt file from a php array.

Is it correct or should I create an RobotstxtWriter into Sonata/Exporter and use an ArraySourceIterator ?

[greg0ire]

Please avoid naming anything *Manager, it's too vague. What do you think about RobotsTxtGenerator?

Is it correct or should I create an RobotstxtWriter into Sonata/Exporter and use an ArraySourceIterator ?

Don't use sonata exporter for that, I don't think it's its purpose.

Also, are there no robots.txt generator libs we could integrate?

Finally, maybe start by writing the docs, and make a WIP PR with just that. Then write the tests. Then the code.

[Nightbr]

Thanks for your advice, I will do this ;) Add to that, I will create a command RobotstxtGeneratorCommand.php in order to generate the txt file.

I found some robots.txt generator but it is really basic:

• https://github.com/jayhealey/Robots
• https://github.com/jayhealey/Robots/blob/master/src/Healey/Robots/Robots.php

Only parse lib seems to be popular https://github.com/t1gor/Robots.txt-Parser-Class

Ok for Sonata/Exporter but there is a SitemapWriter that's why I have think of this.

[OskarStark]

I like the idea 👍

[greg0ire]

Ok for Sonata/Exporter but there is a SitemapWriter that's why I have think of this.

Oh wow didn't know that… but sitemaps and robots.txt are different things, right?

[Nightbr]

Yep sure, but I thought we can do the same thing. Source (ArrayIterator) --> Writer (RobotstxtWriter)

But it could be simpler just do it into the SonataSeoBundle

Into the Robotstxt folder creates:

• Robotstxt.php (the main class like this one: https://github.com/jayhealey/Robots/blob/master/src/Healey/Robots/Robots.php)
• RobotstxtGenerator.php (the class which generate the robots.txt using a Robotstxt object)

What do you think ?

[greg0ire]

You're not exporting anything, so IMO that using sonata exporter does not make much sense… I'd go the simple way.

[Nightbr]

All right :ok_hand: Thanks for your help ;) I'm on it!

[OskarStark]

IMO we can have a dedicated RobotsTxtWriter in SonataSeoBundle

[greg0ire]

@OskarStark what data would we be exporting? It's not data here, it's configuration only, right?

[Nightbr]

I think I can rename the RobotsTxtGenerator.php to RobotsTxtWriter.php. In fact, robots.txt is a file we have to create in the web folder.

[greg0ire]

I think we risk confusion with exporters writers if you do that.

[Nightbr]

ok, I keep RobotsTxtGenerator.php

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.