nuxt-modules / robots

Tame the robots crawling and indexing your Nuxt site.
https://nuxtseo.com/robots
MIT License
424 stars 34 forks source link

[Question] How to pass multiple options? #97

Closed kwiat1990 closed 4 months ago

kwiat1990 commented 1 year ago

Hey, I have multiple options for different bots, which look like this:

 User-agent: Googlebot
    Disallow: /nogooglebot/

    User-agent: *
    Allow: /

    Sitemap: `${process.env.CMS_URL}/sitemap/index.xml`

I wonder how I could mirror this file with the robots plugin? I have tried using an array for the rules but it returns only the last entry:

// config
robots: {
    rules: [
      {
        UserAgent: "Googlebot",
        Disallow: "/nogooglebot/",
      },
      {
        UserAgent: "*",
        Allow: "/",
      },
      { 
        Sitemap: `${process.env.STRAPI_URL}/sitemap/index.xml`
      },
    ],
  },

// robots.txt page:
User-agent: *
Disallow: 
abyshake-anand commented 1 year ago
robots: [
    {
        UserAgent: 'Googlebot',
        Disallow: [
            '/nogooglebot/'
        ]
    },
    {
        UserAgent: '*',
        Allow: '/',
    },
]
kwiat1990 commented 1 year ago

@abyshake-anand the output on the robot.txt is the same and contains only the last object:

User-agent: *
Disallow: 
abyshake-anand commented 1 year ago

Can you share your complete config that relates to robots? This is the output I get when I use the following config.

modules: [
  // ....
  '@nuxtjs/robots',
  // ....
],
robots: [
  {
    UserAgent: 'Googlebot',
    Disallow: ['/user', '/admin'],
  },
  {
    UserAgent: '*',
    Disallow: '/admin',
  },
],
image
kwiat1990 commented 1 year ago

I have tried with the very same config but got the output you can see in my previous message. I’ve also restarted dev server and even change user agent in Chrome to googlebot but it didn’t make any difference at all.

yauri-io commented 1 year ago

Sorry for messing the PR history.