robots.txt - make i18n optional for specific routes

[ronenteva]

Clear and concise description of the problem

While automatic i18n integration is great, it's not needed for some routes. For example, there's no need to block /fr/cdn-cgi/

Suggested solution

add i18n config that is enabled by default.

export default defineNuxtConfig({
  robots: {
    disallow: ['/secret', '/admin'],
    i18n: false
  }
})

Alternative

No response

Additional context

No response

[harlan-zw]

In Nuxt Simple Robots 4.0.0-rc.15 you have two options:

1. Opt-out of all i18n locale splitting:

export defineNuxtConfig({
  robots: {
    autoI18n: false,
  }
})

2. Opt-out of i18n locale splitting for a group:

export defineNuxtConfig({
 robots: {
    groups: [
      {
        disallow: [
          '/docs/en/v*',
          '/docs/zh/v*',
          '/forum/admin/',
          '/forum/auth/',
        ],
        _skipI18n: true,
      },
    ],
  },
})

Let me know if you have any issues.

[ronenteva]

@harlan-zw Thanks! I see that you've automatically excluded /cdn-cgi which is great, I suggest also excluding /api