SEO/Search Ranking Issue Caused by Middleware on Production

ahtashamabbasse commented 1 year ago

[x] Review the documentation: https://clerk.com/docs
[x] Search for existing issues: https://github.com/clerkinc/javascript/issues
[x] Go through package changelog files.
[x] Provide the Frontend API key from your application dashboard.

Package + Version

[x] @clerk/nextjs

Dependencies + versions

{
  "name": "clerk-app",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@clerk/nextjs": "^4.21.8",
    "@clerk/themes": "^1.7.5",
    "autoprefixer": "10.4.14",
    "next": "13.4.7",
    "postcss": "8.4.24",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "tailwindcss": "3.3.2"
  }
}

Browser/OS

If applicable e.g. Chrome 115.0.5790.170, Node v16.20.0

Issue Description:

After following the documentation and successfully deploying the project on the production environment, a significant SEO issue has emerged. The problem primarily affects the landing/home page of the website, which is expected to be accessible without requiring authentication. Despite configuring the project's public routes to include the root ("/") homepage, the issue persists.

Problem Details:

When accessing the homepage under specific conditions, notably when the browser lacks an active indexDB, the page encounters a peculiar behavior. The homepage appears to be re-rendered twice consecutively. The issue can be reproduced by following these steps:

Open Chrome browser. Access Developer Tools. Navigate to the "Application" tab. Under "Storage," clear site data. Refresh the page. Observe that the homepage reloads initially and then redirects.

Impact on SEO:

This behavior is adversely affecting the website's SEO performance, notably on Google search engine results as well as performance evaluation tools like gtmetrix.com and pagespeed.web.dev. The double re-rendering of the homepage, coupled with the redirection behavior, is leading to poor search engine ranking and suboptimal scores on performance evaluation platforms.

Steps to Reproduce:

Clear site data in Chrome's Developer Tools. Refresh the homepage. Note the double re-rendering and subsequent redirection. Expected Behavior: The homepage should load only once, without any redirection, even in scenarios where indexDB is not present in the browser.

Additional Information:

Project follows the documented steps for deployment. Public routes are correctly configured to include the root ("/") homepage. Middleware is suspected to be the cause of the double re-rendering and redirection.

Middleware:

import { authMiddleware } from '@clerk/nextjs';

export default authMiddleware({
   publicRoutes: ['/', '/api/webhook'],
});

export const config = {
   matcher: ['/((?!.*\\..*|_next).*)', '/', '/(api|trpc)(.*)'],
};

Screenshots:

https://github.com/clerkinc/javascript/assets/25702074/a0c8a043-0e0f-40b2-a897-1045ea405de4

Possible Solution:

Investigate the middleware configuration, specifically its interaction with the homepage routing. It seems the middleware might be triggering the re-rendering and redirection behavior. Verify the middleware's role in this process and make necessary adjustments to ensure the homepage loads only once without redirection.

Upon thorough investigation, it has been identified that the '/((?!...|_next).*)' rule within the match is the root cause of the problem. While removing this rule enables the homepage to function seamlessly, it unfortunately disrupts other essential segments of the website, particularly those involving dynamic imports.

Impact:

This issue is significantly impacting the website's SEO and overall user experience. Immediate attention is required to rectify the problem and restore optimal performance.

Please advise on the best course of action to address this issue effectively. Thank you.

colinclerk commented 1 year ago

Hi @ahtashamabbasse - the behavior you're seeing is to support cross-origin session sharing in development mode. When you deploy to production the SEO issue will resolve.

Lev1ty commented 1 year ago

having the same problem in production environment with logged in pages. when the user is logged in, closes the tab, opens another tab and navigates to root route (which is a public route in the middleware), the initial root page response is 401 showing white screen, followed by redirect settling at 200 the root page. this causes the initial page load to be a white screen (white flash) which is undesirable

jescalan commented 1 year ago

👋 Hey friends! So sorry for the delay in response here.

We understand that the flash of white screen (we call this “interstitial”) is not desirable in many cases. We use it to ensure your server has the latest session data before rendering a page, and it impacts signed-in users on their first-visit-back after closing all tabs. It does not have a negative impact on SEO, since search engines are never signed in and therefore will not trigger this flash.

We are looking into some updates to our architecture that would potentially eliminate the white page flash. We’re very aware of it, but don’t yet have a set timeline for this change.

saurabhsri108 commented 1 year ago

It clearly effects the SEO ranking due to performance impact caused by double reload.

colinclerk commented 1 year ago

Have you set production keys (starting with pklive and sklive)? Crawlers (and anyone in a signed out state) are excluded from this token refresh mechanism in production.

saurabhsri108 commented 12 months ago

Does the production keys remain the same except the change from test to live?

colinclerk commented 12 months ago

No, they're completely different. Open your application in the Clerk dashboard. At the top you'll see a dropdown that shows "Development" - switch that to "Production" and it will walk you through steps to productionize.

clerk / javascript