Keep both / and /locale

[monolithed]

Is there any way to have both / and /locale simultaneously?

The issue is that Google is banning websites for the following redirects:

/DEFAULT_LOCAL -> / / -> /USER_LOCALE

[i18nexus]

A couple questions:

1. If you have the same content on / and /default, wouldn't this be bad for SEO since you have duplicate content on multiple pages?
2. As far as I'm aware, Google bots do not have an accept-language header. Which means when they visit your site at /, no redirection will occur via this library. How would Google know that you have redirection at all?
3. Assuming you're using prefixDefault: false, how would Google know that /default exists? Are you mistakenly including /default in your sitemap?

[monolithed]

@i18nexus,

If you have the same content on / and /default, wouldn't this be bad for SEO since you have duplicate content on multiple pages?

it's possible to leave only one copy – this is Google's recommendation, and Next.js supports this out of the box.

As far as I'm aware, Google bots do not have an accept-language header. Which means when they visit your site at /, no redirection will occur via this library. How would Google know that you have redirection at all?

Google, like all search engines, can detect redirects and even partially execute JavaScript code. Google itself informed me about this issue, pointing out the reasons for the ban.

Assuming you're using prefixDefault: false, how would Google know that /default exists? Are you mistakenly including /default in your sitemap?

Apparently, they parsed the URL and requested it in different ways. It's also possible they visited a page where the language is determined by geolocation, remembered that address, and tried using it with a different IP. The fact is, I see such requests in their analytics. I've temporarily closed off these addresses from indexing, but the problem still persists.

[i18nexus]

This library does not use geolocation or IP to determine the language. It just uses the accept-language header. But I get your point they may have tricky complex ways of determining which pages can potentially redirect.

Can you show me what your i18nConfig looks like so I can have a better idea of your setup.

Can you also provide the pathname of the pages that Google says are not served on Google due to redirect?

[i18nexus]

I might also recommend that the hrefs on your links contain the current locale. If your links are always <Link href="/foo" /> even when viewing a page on a non-default locale page like /es, Google will likely click these links and see they are redirected to /es/foo.

For ideal SEO, it is probably ideal to have a component something like this:

const LinkWithLocale(props) {
  const locale = useCurrentLocale(i18nConfig);
  const href = locale === i18nConfig.defaultLocale ? props.href : `/${locale}${props.href}`;

  return <Link {...props} href={href}>{props.children}</Link>;
}

I'll recommend we add this to our documentation.

[monolithed]

@i18nexus,

This library does not use geolocation or IP to determine the language. I determine the geolocation myself )

Can you show me what your i18nConfig looks like so I can have a better idea of your setup.

// middleware.ts

import {
    type NextRequest,
    NextResponse
} from 'next/server';

import {i18nRouter} from 'next-i18n-router';

import {
    getDefaultLocale,
    locales
} from '@/app/i18n/locale';

import {
    type Headers,
    getSecurityHeaders
} from '@/security/headers';

import {DefaultLocaleCookies} from '@/app/cookies';
import {getDefaultCookieOptions} from '@/sdk/cookie';

const setSecurityHeaders = (response: NextResponse, headers: Headers): void => {
    headers.forEach(([name, value, isDev]) => {
        !isDev && response.headers.set(name, value);
    });
};

const setLocale = (response: NextResponse, value: string): void => {
    response.cookies.set(DefaultLocaleCookies.NAME, value, getDefaultCookieOptions());
};

const middleware = (request: NextRequest): NextResponse => {
    const {geo} = request;
    const securityHeaders = getSecurityHeaders(request);
    const defaultLocale = getDefaultLocale(geo?.country);

    const response = i18nRouter(request, {locales, defaultLocale});

    setLocale(response, defaultLocale);
    setSecurityHeaders(response, securityHeaders);

    return response;
};

export const config = {
    matcher: '/((?!api|static|.*\\..*|_next).*)'
};

export {middleware};

// options.ts

import type {Namespace, InitOptions} from 'i18next';

import {
    Locale,
    defaultLocale,
    locales
} from './locale';

import {Company, Domains} from '@/app/config';

const DEFAULT_NAMESPACE = 'translation';

const getOptions = (locale: Locale = defaultLocale, ns: Namespace = DEFAULT_NAMESPACE): InitOptions => {
    return {
        lng: locale,
        supportedLngs: locales,
        fallbackLng: defaultLocale,
        ns,
        defaultNS: DEFAULT_NAMESPACE,
        fallbackNS: DEFAULT_NAMESPACE
    }
};

export {getOptions};
export type {Namespace};

// i18next.ts (client)
import i18next from 'i18next';
import {initReactI18next} from 'react-i18next';
import resourcesToBackend from 'i18next-resources-to-backend';
import LanguageDetector from 'i18next-browser-languagedetector';
import {locales} from './locale';

import {getOptions} from './config';
import {loader} from './loader';

const SSR = typeof window === 'undefined';

i18next.isInitialized || i18next
    .use(initReactI18next)
    .use(LanguageDetector)
    .use(resourcesToBackend(loader))
    .init({
        ...getOptions(),
        lng: undefined,

        detection: {
            order: ['path', 'htmlTag', 'cookie', 'navigator']
        },
        preload: SSR ? locales : []
    });

// i18next.ts (ssr)

import {type Namespace, createInstance} from 'i18next';
import {initReactI18next} from 'react-i18next/initReactI18next';
import resourcesToBackend from 'i18next-resources-to-backend';

import {getOptions} from './config';
import {loader} from './loader';
import {type Locale} from './locale';

const initI18next = async (locale: Locale, namespace?: Namespace) => {
    const i18nInstance = createInstance();

    await i18nInstance
        .use(initReactI18next)
        .use(resourcesToBackend(loader))
        .init(getOptions(locale, namespace))

    return i18nInstance;
};

type Props = {
    locale: Locale;
    namespace?: Namespace;
    key?: string;
};

// SSR
const useTranslation = async ({locale, namespace, key}: Props) => {
    const i18n = await initI18next(locale, namespace);
    const t = i18n.getFixedT(locale, namespace, key);

    return {t, i18n};
};

export {useTranslation};

// change-locale.ts
'use client'

import {
    useEffect,
    useState,
    Dispatch,
    SetStateAction
} from 'react';

import {
    useParams,
    usePathname
} from 'next/navigation';

import {
    type KeyPrefix,
    Namespace
} from 'i18next';

import Cookies from 'universal-cookie';

import {
    type UseTranslationResponse,
    FallbackNs,
    useTranslation as useTranslationOrg
} from 'react-i18next';

import {useRouter} from '@/components/progress-bar/suspense';
import {LocaleCookies} from '@/app/cookies';

import {
    Locale,
    defaultLocale
} from './locale';

import './i18next';

type Translation = UseTranslationResponse<FallbackNs<string>, KeyPrefix<string>>;

const SSR = typeof window === 'undefined';

type LocaleState<T extends Locale> = [T, Dispatch<SetStateAction<T>>];

const useLocale = <T extends Locale>(): LocaleState<T> => {
    const {locale} = useParams<{locale: T}>();

    return useState(locale);
};

type TranslationProps = {
    namespace?: Namespace;
    key?: string;
};

const useTranslation = ({namespace, key}: TranslationProps): Translation => {
    const [pathLocale] = useLocale();
    const result = useTranslationOrg(namespace, {keyPrefix: key});
    const {i18n} = result;

    const cookies = new Cookies(null, {
        path: <string>LocaleCookies.PATH,
        maxAge: <number>LocaleCookies.MAX_AGE
    });

    if (SSR && pathLocale && i18n.resolvedLanguage !== pathLocale) {
        i18n.changeLanguage(pathLocale);
    }
    else {
        // eslint-disable-next-line react-hooks/rules-of-hooks
        const [language, setLanguage] = useState(i18n.resolvedLanguage);

        // eslint-disable-next-line react-hooks/rules-of-hooks
        useEffect(() => {
            if (language === i18n.resolvedLanguage) {
                return void 0;
            }

            setLanguage(i18n.resolvedLanguage);
        }, [language, i18n.resolvedLanguage]);

        // eslint-disable-next-line react-hooks/rules-of-hooks
        useEffect(() => {
            if (!pathLocale || i18n.resolvedLanguage === pathLocale) {
                return void 0;
            }

            i18n.changeLanguage(pathLocale);
        }, [pathLocale, i18n]);

        const cookieLocale = cookies.get(LocaleCookies.NAME);

        // eslint-disable-next-line react-hooks/rules-of-hooks
        useEffect(() => {
            if (cookieLocale === pathLocale) {
                return void 0;
            }

            cookies.set(LocaleCookies.NAME, pathLocale);

        // eslint-disable-next-line react-hooks/exhaustive-deps
        }, [pathLocale, cookieLocale]);
    }

    return result;
};

type ChangeLocaleState<T extends Locale> = [T, (locale: T) => void];

const useChangeLocale = <T extends Locale>(language: T): ChangeLocaleState<T> => {
    const router = useRouter();
    const {locale: pathLocale} = useParams();
    const pathname = usePathname();
    const [locale, setLocale] = useLocale<T>();
    const date = new Date();

    const dispatcher = (locale: T): void => {
        const maxAge = date.getTime() + LocaleCookies.MAX_AGE;

        date.setTime(maxAge);
        const currentPathname = new RegExp(`^/${pathLocale}`);

        if (language === defaultLocale) {
            const newPathname = pathname.replace(currentPathname, '');

            router.push('/' + locale + newPathname);
        }
        else {
            let newPathname = pathname.replace(currentPathname, `/${locale}`);

            router.push(newPathname);
        }

        setLocale(locale);

        router.refresh();
    };

    return [locale, dispatcher];
};

export {
    useLocale,
    useChangeLocale,
    useTranslation
};

export type {Translation};

Code's a bit messy, but here's everything related to localization, just in case.

Can you also provide the pathname of the pages that Google says are not served on Google due to redirect?

Yep, https://wayhunter.com root page

[i18nexus]

Oh, ok. If you are manually using geolocation then this is your problem. Google is most certainly running into redirects on your app. This library does not use geolocation in the default localeDetector for that reason. Using request headers is a lot more SEO friendly.

When it comes to Google, using redirects for multi-language purposes at all is not recommended. Whether it's via headers or geolocation. They specifically say:

"Avoid automatically redirecting users from one language version of a site to a different language version of a site. For example, don't redirect based on what you think the user's language may be. These redirections could prevent users (and search engines) from viewing all the versions of your site." https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites

So if you are going to use redirects at all, using the accept-language header carefully is certainly the better of the 2 evils. If using headers carefully, Google will not experience your redirects at all since it wouldn't have a way of knowing your redirects exist. Their bots do not use an accept language header:

"In addition, the crawler sends HTTP requests without setting Accept-Language in the request header." https://developers.google.com/search/docs/specialty/international/locale-adaptive-pages

Is use of geolocation an absolute requirement for you?

[monolithed]

Is use of geolocation an absolute requirement for you?

It's not very critical, although it is quite a common and convenient practice. From my point of view, the redirects are set up correctly, but from the search engines' perspective, they are not. As a temporary workaround, I have hidden the default locale from the robots and sent the pages for reindexing. If the situation does not change in 3 days, I will think about how to solve this problem. Thanks for your help.

[i18nexus]

Yeah, I think as long as you're using geolocation, you'll likely continue running into these problems. The bot's IP is always in the the United States as can be read about in the page I linked above. So if your default language is not English, all of your pages will redirect to /en/.... I personally would avoid using geolocation. But hope you the best on figuring it out, good luck!

[i18nexus]

After looking at your code again, I do see a problematic line:

const defaultLocale = getDefaultLocale(geo?.country);

The defaultLocale used with next-i18n-router really should not be dynamic. This means if someone in Spain visits your site, / will be in Spanish and /es won't exist. But to someone in the United States, / will be in English and /en won't exist. This can certainly be adding to your SEO problems.

I see what you're trying to do, and it is a creative way to make it so people see / as their native language. But it's not very conventional, especially to bots.

[monolithed]

The defaultLocale used with next-i18n-router really should not be dynamic. This means if someone in Spain visits your site, / will be in Spanish and /es won't exist. But to someone in the United States, / will be in English and /en won't exist. This can certainly be adding to your SEO problems.

Not quite right. I have a fixed list of 5 locales. If a user visits from Spain, they will see the page in English because /es doesn't exist. I check in advance if I have such a locale. However, if a user visits from a country where French is spoken, like Benin or Wallis and Futuna, I'll show them the text in French because /fr exists.

enum Locales {
    EN = 'en',
    FR = 'fr',
    KK = 'kk',
    RU = 'ru',
    ZH = 'zh'
}

enum Countries {
    FRANCE = 'FR',
    ....
}

const countries = {
    [Locales.FR]: [
        Countries.BENIN,
        Countries.WALLIS_AND_FUTUNA,
        ....
    ],
    ....
};

const defaultLocale = cookies.get(DefaultLocaleCookies.NAME);

const getDefaultLocale = (locale?: string): string => {
    const fallbackLocale = defaultLocale || Locales.EN;

    if (!locale) {
        return fallbackLocale;
    }

    const zones = Object.entries(countries);

    for (const [zone, countries] of zones) {
        if (countries.includes(<Countries>locale!)) {
            return zone;
        }
    }

    return fallbackLocale;
};

This approach is used on many well-known websites, but I don't know how to solve the indexing problem. The problem is that when the bot accesses from France, /fr becomes the default page and a request like /fr redirects to /, but when the bot accesses from the USA, /fr exists and does not redirect to /.

I temporarily disabled geolocation. Microsoft's language switching works similarly, but they don't have such problems with indexing...

[klonwar]

@monolithed What approach did you end up taking?

[monolithed]

@monolithed What approach did you end up taking?

@klonwar, the issue is still relevant for me; I haven’t found a way to keep two paths simultaneously. For now, I had to give up on automatic locale detection to avoid being banned by search engines. Ideally, all paths should be accessible because search engines remember the paths of all locales, and if they can't find one, they exclude the main page from the index. I would reopen this task if it can be solved.