vercel / next.js

The React Framework
https://nextjs.org
MIT License
124.9k stars 26.66k forks source link

Cache-Control headers set in next.config.js are overwritten #22319

Closed hartshorne closed 4 days ago

hartshorne commented 3 years ago

What version of Next.js are you using?

10.0.7, 10.0.8-canary.3

What version of Node.js are you using?

15.8.0

What browser are you using?

curl, Chrome

What operating system are you using?

macOS

How are you deploying your application?

next start

Describe the Bug

Custom Cache-Control headers configured in next.config.js are overwritten in some cases. It looks any page that use getStaticProps will have their cache headers overwritten with Cache-Control: s-maxage=31536000, stale-while-revalidate which seems to come from https://github.com/vercel/next.js/blob/80c9522750a269a78c5636ace777a734b9dcd767/packages/next/next-server/server/send-payload.ts#L34

Expected Behavior

When we configure a Cache-Control header, don't set it to something else.

To Reproduce

Starting from yarn create next-app --example headers headers-app, here's a sample project that demonstrates the bug: https://github.com/hartshorne/headers-app

You can clone the project, and run something like yarn build && yarn start to start it in production mode. (dev mode overwrites all Cache-Control headers to prevent the browser from caching during development, which makes sense.)

Here's the next.config.js: https://github.com/hartshorne/headers-app/blob/main/next.config.js

Here's props.js (which exports getStaticProps) – this is broken: https://github.com/hartshorne/headers-app/blob/main/pages/props.js

% curl -I http://localhost:3000/props 
HTTP/1.1 200 OK
Cache-Control: s-maxage=31536000, stale-while-revalidate
X-Custom-Header: with static props
X-Powered-By: Next.js
ETag: "978-AzRqkosC2YfRQvbfuY7KiKb51e8"
Content-Type: text/html; charset=utf-8
Content-Length: 2424
Vary: Accept-Encoding
Date: Thu, 18 Feb 2021 22:09:41 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Here's static.js — this works: https://github.com/hartshorne/headers-app/blob/main/pages/props.js

% curl -I http://localhost:3000/static     
HTTP/1.1 200 OK
Cache-Control: public, max-age=3600, s-maxage=60, stale-while-revalidate=86400
X-Custom-Header: no props, no link
X-Powered-By: Next.js
ETag: "945-MX0a4VCO/O001kyDcNn9a55taLQ"
Content-Type: text/html; charset=utf-8
Content-Length: 2373
Vary: Accept-Encoding
Date: Thu, 18 Feb 2021 22:09:10 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Note that X-Custom-Header comes through, but the Cache-Control header is overwritten. Same behavior in Chrome, and with a regular GET request.

hartshorne commented 3 years ago

@ijjk, it looks like https://github.com/vercel/next.js/commit/a32b1f487062c2976aa953a81f46069be2584425 introduced code that would overwrite custom Cache-Control headers -- maybe we can check to see if the Cache-Control headers were set before overwriting them?

hartshorne commented 3 years ago

The bug exists in https://github.com/vercel/next.js/releases/tag/v10.0.8-canary.3 as well: https://github.com/hartshorne/headers-app/tree/canary

hartshorne commented 3 years ago

Related: https://github.com/vercel/next.js/issues/19914

hartshorne commented 3 years ago

This would fix the issue: https://github.com/vercel/next.js/compare/canary...hartshorne:preserve-custom-cache-control-headers

But seems to run against the grain of the documentation: https://nextjs.org/docs/api-reference/next.config.js/headers#cache-control

Cache-Control headers set in next.config.js will be overwritten in production to ensure that static assets can be cached effectively. 

@timneutkens reasonable defaults are great, but it seems like there should be a way to easily override something like a Cache-Control header.

FDiskas commented 3 years ago

it works for me on next@10.0.7 - all images served from public directory with cache control header at next.config.js

  async headers() {
    return [
      {
        source: '/:all*(svg|jpg|png)',
        locale: false,
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=9999999999, must-revalidate',
          }
        ],
      },
    ]
  },
ramiel commented 3 years ago

Any news on this? I have an api endpoint that produces images and I want to cache them on the client. Why can't I specify the desired cache-control headers?

When the documentation says

Cache-Control headers set in next.config.js will be overwritten in production to ensure that static assets can be cached effectively.

makes too many assumptions on the user use-case.

FDiskas commented 3 years ago

@ramiel vote for https://github.com/vercel/next.js/pull/23328 :)

ramiel commented 3 years ago

@FDiskas I voted for that one, but it's not exactly the same issue. That is specific for the images through next/image I guess, this is for anything

FDiskas commented 3 years ago

Take a look at my example in the comments, this works on production as well

flockonus commented 3 years ago

This is a considerable unchangeable default can cause pages to be stale up to 1 years for CDNs, which is bonkers(!) And agree with @ramiel , these are too many assumptions about use case, for a config that can't be changed.

Also note that stale-while-revalidate seems to be used being used incorrectly, as it should have =seconds to it

schlosser commented 2 years ago

Big +1 here – when using next/image with images from the /public folder deployed outside of Vercel, we're getting:

cache-control: public, max-age=0, must-revalidate

No matter what we put in next.config.js. This makes our images load extremely slow. We'd like our CDN to use stale-while-revalidate behavior but we can't get our webserver to respect this setting...

cweekly commented 2 years ago

I'm flabbergasted by:

"Cache-Control headers set in next.config.js will be overwritten in production to ensure that static assets can be cached effectively."

Sane defaults would be one thing. But this is just nuts. "Using next.js means you cannot configure your app's HTTP response headers."

Please, maintainers, provide some way to opt out of this heavy-handed, unpredictable, one-size-fits-some behavior.
Please.

yarolegovich commented 2 years ago

Was quite disappointed to find this out. The plan was to have the following setup for a hosted CMS:

I'm using CDN-Cache-Control header to overwrite CDN s-maxage, but it's frustrating we can't control resources headers 👎

jekh commented 2 years ago

Is there truly no way to set your own cache-control headers for statically-generated pages? You can set the s-maxage value by remembering to set revalidate in every return statement within every getStaticProps(), but what if you don't want to use the default s-maxage=X, stale-while-revalidate pattern for the cache-control header?

Would a middleware potentially work for getStaticProps pages? Does that fire for all requests for static pages, even when the page is still fresh? If so, will setting response headers still work with the new middleware?

The default for getStaticProps pages seems like quite the footgun: unlike every other page, it automatically adds a very high default s-maxage -- don't forget, or that page could be stuck in your CDN forever!

imconfused218 commented 1 year ago

Really frustrating. Seeing bugs with ISR pages that will sometimes default to s-maxage=31536000 instead of using their revalidate page setting. Pages that should be re-generating every 2 minutes are suddenly stuck for a year. Please consider setting more appropriate defaults or allow a cache-control or max-age configuration

yangxuHBO commented 1 year ago
async headers() {
    return [
      {
        source: '/:all*(svg|jpg|png)',
        locale: false,
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=9999999999, must-revalidate',
          }
        ],
      },
    ]
  },

My nextJS is v11. Did not work for me. Still got some other max-age rather than this one.

Mihai-github commented 1 year ago

Hi, I'm having the same issue with a nex.js website with a static build hosted on S3. Even though I'm setting in the next.config.js file rule for cache-control:

async headers() {
    return [
      {
        source: '/:all*(svg|jpg|png)',
        locale: false,
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=9999999999, must-revalidate',
          }
        ],
      },
    ]
  },

In network tab I end up with Request Headers cache-control: no-cache

AdamZaczek commented 1 year ago

We've spent a week trying to fix this issue. The app is broken for thousands of CodeAlly users because of this behavior. We use Vercel + Next and migrated from Gatsby. Right now, we can not serve a production fix for our user base because cache is stuck.

We need an option to set the cache header or we can't deliver the product in any shape or form. Please.

u3u commented 1 year ago

Before the #39707 merge, if anyone else is bothered by it, my solution is to modify node_modules/next/dist/server/send-payload/revalidate-headers.js and then use patch-package to generate the patch. If you use pnpm then you can just use pnpm patch next to generate the patch using pnpm patch-commit after making changes in the temporary directory.

AdamZaczek commented 1 year ago

I tried to force revalidation @u3u:

diff --git a/node_modules/next/dist/server/send-payload/revalidate-headers.js b/node_modules/next/dist/server/send-payload/revalidate-headers.js
index 4d9250f..fddad05 100644
--- a/node_modules/next/dist/server/send-payload/revalidate-headers.js
+++ b/node_modules/next/dist/server/send-payload/revalidate-headers.js
@@ -6,15 +6,15 @@ exports.setRevalidateHeaders = setRevalidateHeaders;
 function setRevalidateHeaders(res, options) {
     if (options.private || options.stateful) {
         if (options.private || !res.hasHeader("Cache-Control")) {
-            res.setHeader("Cache-Control", `private, no-cache, no-store, max-age=0, must-revalidate`);
+            res.setHeader("Cache-Control", `private, no-cache, no-store, s-max-age=0, max-age=0, must-revalidate`);
         }
     } else if (typeof options.revalidate === "number") {
         if (options.revalidate < 1) {
             throw new Error(`invariant: invalid Cache-Control duration provided: ${options.revalidate} < 1`);
         }
-        res.setHeader("Cache-Control", `s-maxage=${options.revalidate}, stale-while-revalidate`);
+        res.setHeader("Cache-Control", `s-maxage=${options.revalidate``}, must-revalidate`);
     } else if (options.revalidate === false) {
-        res.setHeader("Cache-Control", `s-maxage=31536000, stale-while-revalidate`);
+        res.setHeader("Cache-Control", `s-maxage=0, max-age=0, must-revalidate`);
     }
 }

This among all the other solutions, did not work for me. Could you share the patch file?

u3u commented 1 year ago

@AdamZaczek You should follow the link above to modify the content:

diff --git a/dist/server/send-payload/revalidate-headers.js b/dist/server/send-payload/revalidate-headers.js
index 4d9250fa1902ecdca0a4220df9863082d49aa00a..b554381956007582e7a5fb678b89e2841a8fee16 100644
--- a/dist/server/send-payload/revalidate-headers.js
+++ b/dist/server/send-payload/revalidate-headers.js
@@ -5,16 +5,20 @@ Object.defineProperty(exports, "__esModule", {
 exports.setRevalidateHeaders = setRevalidateHeaders;
 function setRevalidateHeaders(res, options) {
     if (options.private || options.stateful) {
-        if (options.private || !res.hasHeader("Cache-Control")) {
+        if (options.private || !res.getHeader("Cache-Control")) {
             res.setHeader("Cache-Control", `private, no-cache, no-store, max-age=0, must-revalidate`);
         }
     } else if (typeof options.revalidate === "number") {
         if (options.revalidate < 1) {
             throw new Error(`invariant: invalid Cache-Control duration provided: ${options.revalidate} < 1`);
         }
-        res.setHeader("Cache-Control", `s-maxage=${options.revalidate}, stale-while-revalidate`);
+        if (!res.getHeader("Cache-Control")) {
+            res.setHeader("Cache-Control", `s-maxage=${options.revalidate}, stale-while-revalidate`);
+        }
     } else if (options.revalidate === false) {
-        res.setHeader("Cache-Control", `s-maxage=31536000, stale-while-revalidate`);
+        if (!res.getHeader("Cache-Control")) {
+            res.setHeader("Cache-Control", `s-maxage=31536000, stale-while-revalidate`);
+        }
     }
 }

Then you can override the Cache-Control headers in next.config.js

/** @type {import('next').NextConfig} */
const nextConfig = {
  headers: async () => {
    return [
      {
        headers: [
          {
            key: 'Cache-Control',
            value: 'private, no-cache, no-store, max-age=0, must-revalidate',
          },
        ],

        source: '/:path*',
      },
      {
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=31536000, immutable',
          },
        ],

        source: '/:path(.+\\.(?:ico|png|svg|jpg|jpeg|gif|webp|json|js|css|mp3|mp4|ttf|ttc|otf|woff|woff2)$)',
      },
      {
        headers: [
          {
            key: 'Cache-Control',
            value: 'private, no-cache, no-store, max-age=0, must-revalidate',
          },
        ],

        source: '/_next/data/:path*',
      },
      {
        headers: [
          {
            key: 'Cache-Control',
            value: 'private, no-cache, no-store, max-age=0, must-revalidate',
          },
        ],

        source: '/_next/:path(.+\\.(?:json)$)',
      },
    ];
  },
};
AdamZaczek commented 1 year ago

Thank you @u3u! It appears, our problem is not fixed even with your version of the revalidate0headers file but this might help others.

MartinDavi commented 1 year ago

This is a really bad issue - we're trying to use SSG with on demand ISR. We don't want to use the revalidate prop in getStaticProps because we don't need our next.js server to regenerate pages every n number of seconds. Pages will be regenerated automatically on demand when their content changes using on demand ISR.

But because I can't influence the cache-control header in any other way than using the revalidate prop, I'm unable to set a reasonable s-maxage so that our CDN will phone home every once and a while to get the fresher SSG pages.

This is both a complete head scratcher as to why this isn't configurable, and a deal breaker for the workflow we want to use.

edit:

@jekh

Would a middleware potentially work for getStaticProps pages? Does that fire for all requests for static pages, even when the page is still fresh? If so, will setting response headers still work with the new middleware?

No, this doesn't work - I just tried. I'm able to set custom response headers in the middleware, but trying to set Cache-Control header gets overwritten down the line. Great idea, though.

Rototu commented 1 year ago

@u3u Your solution worked perfectly for me. However I still find the fact that patching next.js package is the only method to do overwrite SSG/ISR headers a huge downside of using Next.js.

For multiple clients I've worked with (giant e-commerce clients) this always proved to be a major headache.

In one instance we completely abandoned SSG and just used SSR with caching handled at CDN level.

Encountering same issue with another client now, trying solution above, but it's crazy that this is the only solution available if we want to keep SSG.

Many organisations I've worked with would be much happier to abandon Next.js than to abandon their current CDN and Cloud providers in favour of Vercel.

Therefore, the value added by enabling custom headers therefore would be huge for both users and for the adoption of Next.js, at least from my experience 😃 .

amannn commented 1 year ago

My use case was to modify the cache-control header for SSR requests with the App Router.

Since there's no official way to achieve this and headers set within next.config.js for these paths are ignored, I had no other choice than to set them in revalidate-headers.js:

diff --git a/dist/server/send-payload/revalidate-headers.js b/dist/server/send-payload/revalidate-headers.js
index eae390d4927601057e38e95d6cda615342e9674e..b83df25eebfee4db68049f8bdf267dd0cd22dd2c 100644
--- a/dist/server/send-payload/revalidate-headers.js
+++ b/dist/server/send-payload/revalidate-headers.js
@@ -10,9 +10,14 @@ Object.defineProperty(exports, "setRevalidateHeaders", {

 });
  function setRevalidateHeaders(res, options) {
       if (options.private || options.stateful) {
-        if (options.private || !res.getHeader("Cache-Control")) {
-            res.setHeader("Cache-Control", `private, no-cache, no-store, max-age=0, must-revalidate`);
-        }
+        if (options.private) {
+            res.setHeader("Cache-Control", `private, no-cache, no-store, max-age=0, must-revalidate`);
+        } else {
+            // IMPORTANT: This matches all dynamic content routes and applies the same 
+            // caching headers (10min CDN caching with infinite stale-while-revalidate).
+            // You can also use `res.req.url` to match specific routes.
+            res.setHeader("Cache-Control", "public, s-maxage=600, stale-while-revalidate=31557600");
+         }
     } else if (typeof options.revalidate === "number") {
               if (options.revalidate < 1) {
                       throw new Error(`invariant: invalid Cache-Control duration provided: ${options.revalidate} < 1`);
cdpark0530 commented 1 year ago

It's very strange to me that Vercel, the mother of Next.js, says two things contradictory to each other. This Vercel page guides how to configure headers in next.config for ISR, while this Next.js document notes that Cache-Control headers in next.config will be overwritten in production.

I applied @u3u's patch, and I saw the header applies well on production mode locally, but this patch creates another issue on Vercel platform, and it wasn't solved in v13.3.1-canary.0 unlike this says.

(I don't know since when but at least) on next@13.4.4, it seems that Cache-Control headers set from headers in next.config are always removed before setRevalidateHeaders is excuted, so res.getHeader("Cache-Control") in the function is always undefined. Nasty workaround would be setting Cache-Control header in different name in next.config and then checks it in setRevalidateHeaders, but surely Vercel/Next.js should give us a way to make on-demand ISR

matolink commented 1 year ago

August 2023 and still nothing, please Vercel! this is a must for many use cases!

FDiskas commented 1 year ago

https://github.com/vercel/next.js/pull/27200

darl0ck commented 1 year ago

@timneutkens Check pls this

darl0ck commented 1 year ago

They break it again in 13.4.13, In 13.4.12 all works fine If i set appDir:false, when using Page Dir, after 13.4.12 this hotfix doesnt work at all (((

yhakbar commented 11 months ago

Hello! Currently, this is impairing our ability to adopt NextJS effectively, as there is no way to host an app that uses Header based routing without using something like NGINX to override the headers returned by a NextJS app.

I detailed this issue at some length here in the NextJS Discord: https://discord.com/channels/752553802359505017/1149432481372717116

It would be great if an escape hatch was provided for this kind of functionality so that 1:1 migrations from other application frameworks like Django and Ruby on Rails were more feasible, as they can explicitly set the Cache-Control response header.

While certain edge cases can be addressed through the use of middleware and dynamic routes, it feels like a very indirect way to almost get the same functionality that's possible just by explicitly setting response headers to desired values when necessary.

cbratschi commented 11 months ago

I had the same issue and this worked for me: https://github.com/vercel/next.js/discussions/20991#discussioncomment-7166243

@yhakbar using NGINX too. I would prefer an additional option for the public folder assets. Route based filters might have some side effects.

leerob commented 10 months ago

I'm working on documentation here to better explain the automatic caching headers that are added: https://github.com/vercel/next.js/pull/58027

joekur commented 10 months ago

If you set revalidate: false, it will default to a one-year cache duration.

@leerob how is this supposed to work when you want to deploy a new version of your application? I don't know if vercel is doing something special, but if you're self hosting, and you have a CDN in front of your app, how are you expected to be able to ever make updates?

My best guess after looking at vercel's architecture is that the cache is somehow specific to a particular deploy, whereas when self hosting the cache will simply be based on the requested URL?

This is a major issue for our team that self hosts next.js with a cloudfront distribution.

Karnak19 commented 10 months ago

I want to relate, maybe I'm doing it wrong somewhere, but serverless executions are going out of hand (and my boss is not happy 💰 ) due to this.

export const getStaticProps = hydratedGetStaticProps(async (context: GetStaticPropsContext) => {
  const params = context.params as Params<{ categoryId: string }>;

  const queryClient = new QueryClient();

  await queryClient.prefetchQuery({
   // ...
  });

  return {
    props: {
      dehydratedState: dehydrate(queryClient),
    },
    revalidate: 1200,
  };
});

export const getStaticPaths: GetStaticPaths = async () => {
  return {
    paths: [],
    fallback: 'blocking',
  };
};
export const hydratedGetStaticProps =
  <P extends { dehydratedState?: ReturnType<typeof dehydrate> } = any>(getStaticProps: GetStaticPropsCallback<P>) =>
  async (context: GetStaticPropsContext) => {
    const pageLocale = context.locale ?? context.defaultLocale ?? env.NEXT_PUBLIC_DEFAULT_LOCALE;

    const headers = { 'x-account-key': env.NEXT_PUBLIC_ACCOUNT_KEY };

    const queryParams = { language: pageLocale };

    const queryClient = new QueryClient();

    const [translations, gsp] = await Promise.all([
      getServerTranslations(pageLocale),
      getStaticProps(context),
      queryClient.prefetchQuery({
        // ...
      }),
    ]);

    if (!('props' in gsp)) {
      return gsp;
    }

    return {
      revalidate: 900,
      ...gsp,
      props: {
        ...translations,
        ...gsp.props,
        dehydratedState: {
          mutations: [...((gsp.props.dehydratedState ?? {})?.mutations ?? []), ...(dehydrate(queryClient).mutations ?? [])],
          queries: [...((gsp.props.dehydratedState ?? {})?.queries ?? []), ...(dehydrate(queryClient).queries ?? [])],
        },
      },
    };
  };

Capture d’écran 2023-11-07 à 10 05 35

yhakbar commented 10 months ago

Cross posting comment here, as I didn't realize I was making it on a non-NextJS repo earlier:

Does anyone know why it is that this functionality isn't exposed as an opt-in configuration in the app router for dynamic pages?

There are business use-cases that benefit from being able to explicitly set Cache-Control headers to desired values, even when the pages are dynamically generated on-the-fly.

While there might be a workable solution that has the page statically rendered or statically rendered with ISR, and that solution might be more efficient than just setting the Cache-Control header to a desired value explicitly for a dynamically rendered page, why remove that flexibility?

ISR doesn't allow for continuously updated dynamic routes, as far as I know, and it seems like the only alternative to having the ability to explicitly set Cache-Control for situations like this is to either do some complicated logic with middleware, or use a reverse proxy to inject the desired Cache-Control headers after the response leaves NextJS.

iddk0321 commented 10 months ago

I deployed my app using the standalone option to create a server.js file, and then deployed it using Docker. I'm not sure if this is the best method, but it works as I want, so I'm sharing it.

When building with the Dockerfile, add the following command:

RUN sed -i 's/public, max-age=31536000, immutable/no-cache, no-store, must-revalidate/' ./node_modules/next/dist/server/next-server.js

The 'sed' command is similar to JavaScript's replace, which finds the text you want and replaces it with another. In the node_modules/next/dist/server/next-server.js file, you can see that the Cache-Control header is overridden as 'public, max-age=31536000, immutable'. (Search for the setImmutableAssetCacheControl function.)

You can change this code to the cache mode you want. I changed it to use the 'no-cache, no-store, must-revalidate' option.

Below is my complete Dockerfile.


FROM node:18-alpine AS base

FROM base AS deps

RUN apk add --no-cache libc6-compat
WORKDIR /app

COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* ./
RUN \
  if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
  elif [ -f package-lock.json ]; then npm ci; \
  elif [ -f pnpm-lock.yaml ]; then yarn global add pnpm && pnpm i --frozen-lockfile; \
  else echo "Lockfile not found." && exit 1; \
  fi

FROM base AS builder
WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules

COPY . .

RUN yarn build

FROM base AS runner
WORKDIR /app

ENV NODE_ENV production
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/public ./public

RUN mkdir .next
RUN chown nextjs:nodejs .next
RUN yarn global add pm2
RUN npm install sharp

COPY ecosystem.config.js ./ecosystem.config.js
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

USER nextjs

EXPOSE 9300
ENV PORT 9300
ENV HOSTNAME "0.0.0.0"

RUN sed -i 's/public, max-age=31536000, immutable/no-cache, no-store, must-revalidate/' ./node_modules/next/dist/server/next-server.js

CMD ["pm2-runtime", "start", "ecosystem.config.js"]
robhrt7 commented 9 months ago

RE: @joekur

@leerob how is this supposed to work when you want to deploy a new version of your application? I don't know if vercel is doing something special, but if you're self hosting, and you have a CDN in front of your app, how are you expected to be able to ever make updates?

My best guess after looking at vercel's architecture is that the cache is somehow specific to a particular deploy, whereas when self hosting the cache will simply be based on the requested URL?

This is a major issue for our team that self hosts next.js with a cloudfront distribution.

It does look that Next.js cache headers approach is mainly implemented based on what is true on Vercel platform, as is biased in some way or another. It's indeed an important point that Vercel does invalidate cache much better across deploys, being a highly tailored hosting solution for Next.js, while other CDN services are much more generic purpose and in some cases implementation specifics get in a way of what Next.js output as default.

As different CDN vendors implement their infra differently, and many features that present in the HTTP spec, are either implemented differently across the board, or not supported at all, it's very hard to navigate this topic for different circumstances.

I do wish that Next.js power users could have an ability to override cache headers, and I'm continuously surprised how is this not addressed over the years people been facing these issue. I guess the % of people who care about efficient caching on self-hosted Next.js is still too small and it's easy to get lost in the topic. But with more enterprise adoption, I'm sure Next.js will be forced to give more controls, as I don't see any reason not doing this.

I'm currently working on some articles to cover some Next.js self host issues, to be covered in FocusReactive blog. Hoping to get to the bottom of this at some point.

Chipping in on the solutions options on how things stand right now (what I'll also cover in articles in more detail) - you can partially override some cache headers, for example for fully static pages without data fetching, despite what Next.js docs state. And the option we chose is to add CDN specific cache control headers that are respected by Fastly over the default cache control - Surrogate-Control (note that it's a proprietary header for Fastly)

joekur commented 9 months ago

Not ideal, but we ended up working around this limitation by using a custom server and monkey-patching the http writeHead method. This method seemed slightly less fragile compared to overwriting next's source more directly. The constraints were that Next would overwrite any Cache-Control header already set, and by the time you call handleNextRequest(), the response will already have been written back to the request. So before handing off to handleNextRequest():

  const originalWriteHead = res.writeHead;

  res.writeHead = (...args) => {
    res.setHeader(
      'Cache-Control',
      'private, no-cache, no-store, max-age=0, must-revalidate',
    );

    return originalWriteHead.apply(res, args);
  }

Of course if Next ever changes their implementation to not call writeHead, this will break 😞

This solution fully turns off CDN and local caching for all HTML requests, which for us 1. ensures that middleware will always get run, 2. removes some of the undesirable caching behavior when you mix ISR with a CDN cache, and 3. fixes the issue where static routes were getting cached for a full year causing newly deployed changes to never be seen.

calvinl commented 8 months ago

running across this issue while looking for a way to cache with origin headers in Google's Cloud CDN, which requires 'Cache-Control' to be set with maxage. Really disappointing this hasn't been addressed.

ConnorLanglois commented 8 months ago

Just coming across this bug myself - this seems like a major issue in such a "high profile" framework like NextJS. Honestly, it is very concerning that this wasn't addressed years ago, as this affects anybody self-hosting with truly static pages (i.e. not using revalidate). Anytime you push a new update to your server, the changes won't actually take place since the cache-control headers have caused intermediary cdn's (and proxies!!) to cache the prior version. Major L.

Of course, one can just purge their cdn after updating their server (and hopefully your cdn hasn't been forwarding your cache-control headers...), but what if one doesn't have a cdn sitting in front of their server? Then anybody accessing your site through a proxy will receive old pages for a whole entire year (since s-maxage applies to any shared caches, and a proxy is a shared cache). Wild that this issue actually exists in this framework, I am bewildered that this is the default behavior, and that there is no way to opt-out of it.

dkpoult commented 7 months ago

Same issue as others, we noticed our GCP Cloud Run + CDN site was breaking after we deployed changes because the HTML had references to code that was no longer available. Eventually realised that CDN was caching our statically generated home page (for up to a year??) and that when we rebuild the site on deploy it would cause some file hashes to be change and 404s from the stale HTML trying to fetch them anyways.

Our solution so far has been an entirely unnecessary switch from SSG to SSR because there's just no practical way to override the cache header without us migrating our infrastructure. This is honestly such a mind boggling decision by the Vercel team, and at this point the super opinionated roadblocks that I've hit in adopting NextJS have been enough for me to start recommending against it to other devs considering it.

lhguerra commented 6 months ago

@joekur where did you apply this code? Are you using app or pages router?

joekur commented 6 months ago

@joekur where did you apply this code? Are you using app or pages router?

@lhguerra in a custom server, before handing off the request to the app.getRequestHandler().

lhguerra commented 6 months ago

Hm that's what I was afraid of, this won't work in app router :/

flyjeray commented 5 months ago

Thank you @u3u! It appears, our problem is not fixed even with your version of the revalidate0headers file but this might help others.

@AdamZaczek Answering a year later, but maybe it will be useful for you or the others :)

You might have (or might have not) been confused, but Cache-Control is also overridden for assets (CSS, JS, Fonts, Images..) on the dev server, which will ignore the fix by @u3u. Noting it down here for no one to spend their time on looking for a fix.

It can be fixed by extending patch to include development config:

--- a/node_modules/next/dist/server/dev/next-dev-server.js
+++ b/node_modules/next/dist/server/dev/next-dev-server.js
@@ -832,7 +832,9 @@ class DevServer extends _nextServer.default {
         return await (0, _loadComponents).loadDefaultErrorComponents(this.distDir);
     }
     setImmutableAssetCacheControl(res) {
-        res.setHeader("Cache-Control", "no-store, must-revalidate");
+        if (!res.getHeader("Cache-Control")) {
+            res.setHeader("Cache-Control", "no-store, must-revalidate");
+        }
     }

While the patch above will fix the issues in production, this one will do the thing during development

Note: version of NextJS used is 12.2.5

eloisetaylor5693 commented 5 months ago

Still experiencing this issue in v14 https://github.com/vercel/next.js/issues/62294

ccchapman commented 5 months ago

We are finding this a major hurdle that we cannot control the Cache-Control header from our Next.js SSR application.

6stringbeliever commented 4 months ago

Allow me to add to the growing chorus of people affected by this issue. Looking to implement a solution similar to @joekur above, but truly disheartening to see how many people are facing this issue and the lack of response from Vercel, particularly with an active pull request.

ccchapman commented 4 months ago

Below is a workaround which will let you set a header from Next.js middleware and then rewrite the Cache-Control header from nginx configuration.

function cacheControlMiddleware(): NextResponse | undefined {
  const response = NextResponse.next();
  response.headers.set(
    "X-Cache-Control-Override",
    "public, max-age=60, s-maxage=60, stale-while-revalidate=3600, stale-if-error=86400",
  );
  return response;
}

http {
    map $upstream_http_x_cache_control_override $cache_control_override {
        default $upstream_http_cache_control;
        "~.+?" $upstream_http_x_cache_control_override;
    }

    server {
      # ...

      location / {
          proxy_pass http://127.0.0.1:3000;
          # ...
          proxy_hide_header Cache-Control;
          proxy_hide_header X-Cache-Control-Override;
          add_header Cache-Control $cache_control_override;
      }
}