Build dynamic robots.txt

[yaniv51]

I've found this module as simple and useful, but I think it is missing feature of dynamic robots.txt.

For example, I've 3 domains at the same app. I want that while requesting robots.txt file, it will generate the host of the sitemap.xml dynamically by request host.

Lets assume that we have this robots.txt: User-agent: * Disallow: / Sitemap: {{PROTOCOL}}//{{HOST}}/sitemap.xml

For requesting https://www.example.com/robots.txt, the output for Sitemap will be: Sitemap: https://www.example.com/sitemap.xml And for https://www.example2.com/robots.txt: Sitemap: https://www.example2.com/sitemap.xml

(This feature is available at '@nuxtjs/sitemap' module)

I took a look at the source code, and I think it might be simple for me to implement it.

[ricardogobbosouza]

Hi @yaniv51 Actually, today it is not possible to do that. I will create the option to pass a function in value and so you can customize

[bdrtsky]

How to use it? I don't see related notes in Docs.

[ricardogobbosouza]

@bdrtsky you can pass a function and customize its return.

{
  robots: {
    UserAgent: '*',
    Disallow: () => '/'
  }
}

[ivbstudio]

I try to pass options like this (at the top of nuxt.config.js): let robots = async function() { const res = await axios.get(apiUrl + '/robots') return { res } } and then in module options { robots: robots } But it's not working. Any way to set module options from api call?

[ricardogobbosouza]

Hi @ivbstudio At the moment the module accepts function only in its keys or values ​​and not in the whole object.

[ricardogobbosouza]

@bdrtsky I will update the documentation

[ivbstudio]

Thank you @ricardogobbosouza It will be great, if in the future module will accepts function in the whole object.

[ricardogobbosouza]

@ivbstudio available in v2.4.0

[OndraRehounek]

I couldn't found any mention abot dynamic Sitemap in docs but this is working

robots: {
    UserAgent: '*',
    Allow: '*',
    Sitemap: (req) => `https://${req.headers.host}/sitemap.xml`
},