Closed thedevdavid closed 1 year ago
Similar issue here too on version 13.5.2 and 13.5.3, if you revalidate multiple tags then the behaviour is unreliable. The revalidateTag function seems to succeed, but it is a bit of a black box in what it is actually doing.
A guide to debug on demand revalidation would be great, including a way to visualise what is being cached
Update: I pulled the next.js repo down to have a nosey if there was any way we can debug this. There is an environment variable called NEXT_PRIVATE_DEBUG_CACHE which will set the logging level to debug for caching related stuff (including the revalidateTag implementation). This is the only result I get when I google the var - https://martincapodici.com/2023/06/08/nextjs-undocumented-features/.
From my testing I see this log message being called twice for my two tags that I'm trying to revalidate:
Which then makes an API call to https://lhr1.suspense-cache.vercel-infra.com/v1/suspense-cache/revalidate?tags=${tag}
Which at this point makes me think there's nothing more I can investigate. I'm not sure the code that is running on the Vercel infra is public?
To me it is almost like the revalidateTag function should be accepting an array of tags rather than a singular tag. (even the API endpoint above uses the plural tags
in the query).
@leerob any way you can give some pointers on how I could debug this further?
I have the same, unpredictable, issue as @matthewwilson points out (revalidate a number of tags in a loop).
I tried adding a delay of 100ms
for (const tag of REVALIDATE_TAGS) {
await new Promise((resolve) => {
setTimeout(() => {
revalidateTag(tag);
resolve(true);
}, 100);
});
}
At first it worked out pretty good, but... then it went back being unpredictable.
I tried adding a delay of 100ms
for (const tag of REVALIDATE_TAGS) { await new Promise((resolve) => { setTimeout(() => { revalidateTag(tag); resolve(true); }, 100); }); }
At first it worked out pretty good, but... then it went back being unpredictable.
I tried adding a delay also but it didn't work for me
Started investigating with the Sanity team here: https://github.com/sanity-io/next-sanity/issues/639
According to their investigation, the issue should be on Vercel/Next.js side.
I've been trying out a bunch of things over the past few days that I felt could have been related:
outputFileTracingExcludes
and fetchCache
Still no luck. I've raised a Vercel support case but haven't had a reply yet.
@thedevdavid have you seen that this Vercel repo is awaiting
all of the revalidateTag calls - https://github.com/vercel/platforms
Didn't fix my issue but was wondering if it was something you have tried? The revalidateTag function definition doesn't return a promise but it's kinda weird that Vercel developed code is using await.
I have a very similar problem, but I'm not using sanity
. When I call revalidateTag
, the revalidation is done, but when I return to the page that had the fetch revalidated, the new data is most of the time not updated. Using npm run start
ends up doing the same thing, but when the data is displayed, it is not the most recent.
Out of curiosity, in the Link
element that leads to the page that should have the updated data, I applied the prefetch
property to false
and incredibly everything happens as expected.
I'm also seeing this issue when calling revalidateTag
from a server action, that should theoretically cause a server action refresh if the rendered tree depends on a cache item with that tag.
Did some light debugging with @mwritter, we were specifically seeing issues with tags ending in /
.
The initial re-validation would work but the subsequent ones wouldn't or were off by one.
My guess is that this might apply to tags that include characters that can be url-encoded. There might be a mismatch in what gets processed during the revalidation and non-encoded tags that get saved in-memory.
Also from type signature of revalidateTag
, it's unclear if async / await
is needed.
I think it's worth including given these docs look slightly different
In the current version of the codebase, await
will do nothing, as the revalidateTag
function does not return a Promise
.
However, it does call an async method on the internal incrementalCache
instance (it just doesn't wait for the response):
Thanks for the info @araphiel & @sam3d
To be honest my current feeling is that the issue lies in the Vercel data cache. The code seems to be doing what it is supposed to, up until it calls into Vercel. There are so many different revalidation issues on Github and Discord for a feature that is supposed to be a selling point of Next.js 😢
This definitely seems like a problem on the Vercel side infra. I'm thinking about circumventing the issue by keeping track which revalidation tags should have been revalidated and retrying (by calling revalidateTag()
) until the page's HTTP response age
header is less than what is was when the revalidateTag()
was originally called.
Usually I'm revalidating multiple tags at the same time which seems the flakiest. Manually clearing a single tag seems to work most times.
Example:
revalidateTag([
"model:page-slug",
"frontpage",
"navigation"
])
And as others pointed out, I'd really prefer to await
for a Promise until the revalidation either succeeds or fails rather than fail silently and try to hack my way trough this.
This definitely seems like a problem on the Vercel side infra. I'm thinking about circumventing the issue by keeping track which revalidation tags should have been revalidated and retrying (by calling
revalidateTag()
) until the page's HTTP responseage
header is less than what is was when therevalidateTag()
was originally called.Usually I'm revalidating multiple tags at the same time which seems the flakiest. Manually clearing a single tag seems to work most times.
Example:
revalidateTag([ "model:page-slug", "frontpage", "navigation" ])
And as others pointed out, I'd really prefer to
await
for a Promise until the revalidation either succeeds or fails rather than fail silently and try to hack my way trough this.
I agree, it seems to be more flaky when trying to revalidate multiple tags, and being able to await
the revalidation would be nice, I think for error handling all we can do is wrap revalidateTag
call in a try catch
. 🤔 I don't think revalidateTag
accepts an array
of strings
though (in your example). The docs show the revalidateTag
signature as
revalidateTag(tag: string): void;
I don't think
revalidateTag
accepts anarray
ofstrings
though (in your example). The docs show therevalidateTag
signature asrevalidateTag(tag: string): void;
Yep it sure doesn't. What I meant I usually have a list of tags that I try to revalidate at the same time i.e.
const tagsToRevalidate = [
"model:page-slug",
"frontpage",
"navigation"
]
If you ignore the argument signature, it technically has partial support for multiple string arguments.
It's also not clear why the Vercel example repos are using revalidateTag with async / await.
I also have a problem with revalidateTag which is executed in api chandler and which is triggered by Hygraph CMS webhook. Aplication I have deployed on Vercel. Sometimes revalidation finish well but sometimes I recieve following error:
Failed to revalidate tag pages TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11576:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async globalThis.fetch (/var/task/node_modules/.pnpm/next@13.5.4_@babel+core@7.23.0_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:14:33901)
at async FetchCache.revalidateTag (/var/task/node_modules/.pnpm/next@13.5.4_@babel+core@7.23.0_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/incremental-cache/fetch-cache.js:96:25)
at async Promise.all (index 0) {
cause: ConnectTimeoutError: Connect Timeout Error
at onConnectTimeout (node:internal/deps/undici/undici:8522:28)
at node:internal/deps/undici/undici:8480:50
at Immediate._onImmediate (node:internal/deps/undici/undici:8511:13)
at process.processImmediate (node:internal/timers:476:21)
at process.callbackTrampoline (node:internal/async_hooks:130:17) {
code: 'UND_ERR_CONNECT_TIMEOUT'
}
}
{
operation: 'publish',
data: {
__typename: 'Page',
id: 'clmdolecb7gb00aw6e34j79vi',
stage: 'PUBLISHED'
}
}
isValid true
revalidateTag(TAGS.pages);
revalidation done
Below my api handler used to revalidateTag:
import type { NextRequest } from "next/server";
import { revalidateTag } from "next/cache";
import { verifyWebhookSignature } from "@hygraph/utils";
import { env } from "@/lib/env.mjs";
import { TAGS } from "@/lib/constants";
export async function POST(request: NextRequest) {
const gcmsSignature = request.headers.get("gcms-signature");
if (!gcmsSignature) {
return new Response("Forbidden", { status: 403 });
}
const json: unknown = await request.json();
console.log(json);
if (
!json ||
!(json instanceof Object && "data" in json) ||
!(json.data instanceof Object && "__typename" in json.data)
) {
return Response.json({ status: 204 });
}
const isValid = verifyWebhookSignature({
body: json,
signature: gcmsSignature,
secret: env.HYGRAPH_WEBHOOK_SECRET,
});
console.log("isValid", isValid);
if (!isValid) {
return Response.json({ status: 204 });
}
switch (json.data.__typename) {
case TAGS.cart:
revalidateTag(TAGS.cart);
console.log("revalidateTag(TAGS.cart);");
break;
case "Category":
revalidateTag(TAGS.categories);
console.log("revalidateTag(TAGS.categories);");
break;
case "Page":
revalidateTag(TAGS.pages);
console.log("revalidateTag(TAGS.pages);");
break;
case "Product":
revalidateTag(TAGS.products);
console.log("revalidateTag(TAGS.products);");
break;
}
console.log("revalidation done");
return new Response(null, { status: 204 });
}
Link to repo on GH: https://github.com/grzegorzpokorski/next-shop/blob/main/src/app/api/revalidate-tag/route.ts
Sometimes even if I did not receive error in log tab on Vercel, the page does not update after revalidate.
I've been working on a stripped back example project and have encountered the issue reported by @javadev-jef.
I have an app with 2 pages that show the same data. On initial build the homepage and child page show the correct data.
I then call revalidateTag
The homepage shows the correct data, child page is out of date, even after multiple refreshes
Disabling prefetch on the Link component resolves this issue.
I'm not sure if this is related to the problems I am seeing on our production apps, but will keep digging
It feels like there is a race condition between the actual data cache and the ISR cache. If I revalidate a tag and then immediately refresh the related page, it will have the REVALIDATED
header set, but the actual data is old. If I instead wait a few seconds before refreshing, it will have the same header, REVALIDATED
, but with the new data. I don't know; perhaps the ISR cache is released before the underlying request cache is released. In that case, it should be the opposite.
I'm revalidating several unique tags in a loop, and the fetch method of all requests is POST (GraphQL). Before the test (not tested enough), I purge the data cache (Project => Settings => Data cache => Purge Everything).
After some further testing, I am no longer able to replicate this problem. It's possible that unique tags resolved the issue, or perhaps Vercel underwent some updates. I'm not sure about the exact cause.
However, I haven't yet tested the use of a single tag or multiple shared tags to see if the issue reoccurs.
After some further testing, I am no longer able to replicate this problem. It's possible that unique tags resolved the issue, or perhaps Vercel underwent some updates. I'm not sure about the exact cause.
However, I haven't yet tested the use of a single tag or multiple shared tags to see if the issue reoccurs.
I am experiencing the same. I think 13.5.4 and https://github.com/vercel/next.js/pull/55978 may have resolved the issue.
Everything works for me in 13.5.4. Until I add a page that uses the edge runtime.
My test project structure is:
home page child page edge page
The home page and child page call an API that returns a timestamp. The edge page makes no API calls at all.
When the "edge page" is configured to use the edge runtime, the home page and child page display different timestamps returned from the API, this is despite the fact that caching is enabled, in my opinion both pages should display the same data.
When revalidating by tag, only one of the pages will display the updated information from the API, the other page will never revalidate, despite calling revalidate tag multiple times
Switching the edge page to the node runtime results in all pages displaying the exact same timestamp after a build, meaning the request is being memoized and cached correctly.
When revalidating by tag, all pages revalidate as you would expect
I also have this issue, the revalidateTag sometime not working on some specific tag, I have two pages under /checkout/[page]
/checkout/pageA, fetch data with tag pageA /checkout/pageB fetch data with tag pageB
revalidateTag can only work with pageA, but not working on pageB, I have added log and notice, after revalidate, they page did regenerated after first visit, but the fetch request still return the cached response.
I encountered the same issue. My pages were not updated even though the api/revalidate was triggered and a status 200 returned.
I "fixed" it by using only single tag: ["project:${slug}"] instead of ["project", "project:${slug}"], and also by clicking "Purge Everything" in Data Cache. Looking forward to seeing an update from Vercel regarding the issue with multiple tags.
Any update regarding this issue? It's pretty frustrating that we can't fix this issue. Having multiple tags won't work or do anything at all.
I honestly don't understand why there isn't some option to invalidate the data cache during "next build". My CMS is only usually updated once per week, and when it is, a new deployment is triggered, but stale data is being displayed. I would think that normally the data cache should be ignored/invalidated during a build, but that doesn't seem to be the case. Time-based invalidation wouldn't work in this case, since there are the odd times that an error in the CMS update sooner than that "1-week normal update" might occur, resulting in stale data still being displayed.
EDIT:
I don't know why I didn't think of this before, but changing the build script in package.json to:
"build": "rm -rf .next/cache/fetch-cache && next build",
gives me the caching behavior I want with respect to retrieving fresh data only during build.
I was able to get revalidation working by downgrading to next@13.4.19
and, whenever anything changes in my CMS, calling revalidatePath('/')
and revalidateTags('cms')
. On my site, all fetch
calls are tagged with cms
. I often have to press save twice in my CMS, so those revalidation functions are called twice, before the cache is fully revalidated. But then it works—new information is shown on the site after refreshing (no build required).
This is obviously overkill, it shouldn't need called twice, it shouldn't need to revalidate the root layout no matter what, but as a temporary solution it's working for me and doesn't force me to make pages dynamic, which was a no-go because the site is all static.
@jamesvclements have you tried your workaround with next 14?
I've encountered similar issues to those described above with revalidateTags
behaving inconsistently across different deployments. In my setup, I'm triggering revalidation through a webhook from my CMS (Hygraph), and while the Vercel logs show successful 200 responses for the revalidation calls, the actual content updates on the pages are hit or miss.
I've also noticed is that the production deployment and a git branch deployment, despite running the same code, will often display different results. For instance, one page might update with the latest content on the production site, while remaining stale on the git branch deployment, or vice versa.
I can't reproduce these issues locally running in prod either. I made a POST request (with Bruno) to route handler of the app running locally in production mode and everything works as expected. This leads me to think that the issue is with Vercel.
I suppose that I'm just going to have to periodically trigger a rebuild of my app in the meantime, thankfully the content doesn't update too frequently.
At the moment I'm on Next.js14.0.1
but I've tried downgrading to 13.4.8
with no luck
I have no idea if this discussion is related but I just had a bug with latest next version and one of my api route. Basically I have one api route which return the result of a remote database, nothing else, a simple GET route. My website request to this api route were getting previous data no matter what I did. It was forever stuck on previous data (not my live database). I checked the documentation and found the option fetchCache. Using force-no-store
fixed the issue. I don't understand why nextjs would cache an api route. It makes no sense. Is this a bug ? None of my other api route seems to be cached. It's a mystery why this one specifically behave differently.
Here is my working code as example
import { NextRequest, NextResponse } from "next/server";
import { getSupabaseServerClient } from "../getSupabaseServerClient";
import { getPricingPlans } from "../dbHelpers";
export const GET = async (
_: NextRequest,
{ params }: { params: Record<string, unknown> }
) => {
const client = getSupabaseServerClient();
const { data } = await getPricingPlans(client)();
return NextResponse.json(data);
};
// without this line, the api route is not called anymore and the cached response is returned to the front end
export const fetchCache = "force-no-store";
I am also struggling to understand the concept. In my case, the behavior seems to be consistent when using the same tag for multiple requests.
It seems that a page must be visited at least once after deployment before it works correctly. This means that any pages that have not yet been visited will not be revalidated and will keep the initial content until the tag is revalidated again. So it can happen that some pages show old content and never update to the new content after a fresh deployment. This happens when the content update is timed between the deployment and the first visit.
Let me try to explain with this example.
@martin-coded
Thank you for this comment, I thought I was the only one having this issue. If I have 30 pages I then need to visit them at least once, otherwise revalidation will not work... that's so annoying. Please let me know if you find a solution
I can report the same issue with our project. Using revalidateTag or revalidatePath directly after deployment, before visiting the page at least once does not show new data. On second refresh new data is visible. Running latest canary, hoping for a fix from Vercel soon!
@birdzai Your issue is a bit different from this thread I believe, and what you proposed is the easiest solution for the lack of fetch cache in the build pipeline, just refetching all the data during deployments. However, it is possible to connect to the Vercel Data Cache during deployment with a bit of a hack, cutting out the deluge of fetch
es on every deployment that your solution brings but still providing the most up-to-date data.
The main problem we have is that the deployment build environment isn't given any of the connection environment variables to connect to the Vercel Data cache and use ISR'd data. However, the environment variables are present in API handlers (in order to use revalidateTag
). So, you can actually make a protected API handler that returns those connection strings and load them into the deployment build environment yourself. I've been using this in all my projects recently and it works like a charm.
I made this repo to illustrate the hack / solution: https://github.com/evankirkiles/nextjs-broken-data-cache-demo. See the README for a more detailed explanation.
EDIT: Just to clarify after Javad's comment below, the above isn't a solution for the inconsistent revalidateTag
behavior, which I'm also unfortunately suffering from. It's a fix for a fairly unrelated issue of the build pipeline not using manually-revalidated data.
I can confirm that the behavior mentioned by @martin-coded occurs in a Vercel deployed project with version 14.0.4-canary.3
. It's worth noting that this particular build has resolved numerous cache-related issues.
As it is pointed out, after visiting and manually revalidating each page, calling one revalidatePath("/", "layout")
will trigger a complete project revalidation as intended.
It's important to mention that everything functions smoothly when running npm run start
locally.
Furthermore, I haven't been able to find a method to revalidate a not found
page.
Unfortunately, the solution provided by @evankirkiles only addressed the build cache and didn't resolve this peculiar behavior for me.
Here is my reproduction code: https://nextjs-revalidation-test.vercel.app/ https://github.com/Javad9s/nextjs-revalidation-test
I've had the same problem but using edge functions for the revalidation route by adding export const runtime = "edge";
fixed it for me.
@Haldaug do use any filter in sanity webhook ? Or just add this where fetch sanity with tags ?
@dpalaniichuk I use the edge runtime on the route where I run the revalidateTag(tag)
command. I don't use sanity for my project and have written a simple route handler that gets the tag from the searchParams
of a GET-request.
@Haldaug yep it makes sense then, will try as well
Had the same issues on 14.0.3. Unfortunately I use sanity and their revalidate package uses crypto from nodejs which is not supported in the edge runtime.
On the bright side, I have a small page, so falling back to using revalidatePath("/", "layout")
works fine for my use case.
I've had the same problem but using edge functions for the revalidation route by adding
export const runtime = "edge";
fixed it for me.
Scratch this. This solution worked for a while, but now it's back to the behavior described by OP.
It will undoubtedly work because, currently, with the latest version, setting the runtime edge will disable static regeneration.
I have no idea if this discussion is related but I just had a bug with latest next version and one of my api route. Basically I have one api route which return the result of a remote database, nothing else, a simple GET route. My website request to this api route were getting previous data no matter what I did. It was forever stuck on previous data (not my live database). I checked the documentation and found the option fetchCache. Using
force-no-store
fixed the issue. I don't understand why nextjs would cache an api route. It makes no sense. Is this a bug ? None of my other api route seems to be cached. It's a mystery why this one specifically behave differently.Here is my working code as example
import { NextRequest, NextResponse } from "next/server"; import { getSupabaseServerClient } from "../getSupabaseServerClient"; import { getPricingPlans } from "../dbHelpers"; export const GET = async ( _: NextRequest, { params }: { params: Record<string, unknown> } ) => { const client = getSupabaseServerClient(); const { data } = await getPricingPlans(client)(); return NextResponse.json(data); }; // without this line, the api route is not called anymore and the cached response is returned to the front end export const fetchCache = "force-no-store";
You are instructing the API not to store any cache or data, leading to consistently receiving fresh data. This undermines the intended purpose of caching and the overall usage of revalidateTag.
The tags don't work for me. The only way I have found is by entering presentation mode. I think the cookie "isDraftMode" enter in action and revalidate the cache.
I have encountered the same bug. Right now I am using a server component inside my root layout.js
that uses data from @vercel/kv
. Since the component is shared by different pages, I expected calling revalidatePath("/", "layout")
to update the component on all pages, but only the first page visited after revalidating works. Similarly revalidateTag
also only updates a single page. The rest of the pages continue to show stale data.
More interestingly enough, this only works on Vercel and not on local production builds with pnpm build && pnpm start
Can you try and see if this PR helps? https://github.com/vercel/next.js/pull/58926
We're encountering a similar issue, and it's not specific to Varcel infrastructure. We've deployed this application on Azure in a Linux environment, and we've even experimented with IIS hosting. Unfortunately, it isn't functioning properly. Initially, it worked fine, but suddenly it ceased to operate altogether.
Request you to look into this on priority
Similer Issue https://github.com/vercel/next.js/discussions/58287
It appears that a majority of developers/businesses are encountering this issue.
@vercel-release-bot
also related https://github.com/vercel/next.js/issues/57632
I am running into the same issue on multiple sites. The stack is Strapi + latest NextJS (on Vercel). Weirdly enough even purging the cache in Vercel doesn't clear the old data. RevalidateTag or revalidatePath sometimes refreshes the data but not always and once I get the latest data and rebuild the site I will again see the old data.
Link to the code that reproduces this issue
https://github.com/thedevdavid/productengineerjobs
To Reproduce
Current vs. Expected behavior
I’m trying to get next app router +
revalidateTag()
to work with Sanity but it’s either completely unpredictable or I totally misunderstood something.I now spent 5 full days trying to make it work and build multiple projects. Here’s what I can tell now:
Project 1: This is the most basic, simple setup (posts, pages, categories, [slug] dynamic pages). I think this works. https://github.com/thedevdavid/sanity-revalidate-debug
Project 2: Almost the same setup (posts, pages, categories, [slug] dynamic pages + subpages with extra tags). Here, the simple tags: [
job
] or tags: [benefit
] requests are never getting revalidated on the/
and/post-a-job
routes. But in (for example)/contract/[slug]
route. And the complex tags, like [job:${params.slug}
] or [page:${params.slug}
] are all working well.Projects 3 & 4: Very similar to project 2. posts, pages, and categories with extra schema types, like team members, navigation links, testimonials, and more. Here I can’t even tell the exact problems. Probably similar to project 2 but I’m not sure anymore.
Although there’s 1 thing I know that’s extra. There’s a
homepage
type with the fetch tag,homepage
. This type has the navigation links. There’s a fetch tagged withhomepage
on(site)/page.tsx
and in 2 components (Navbar
&Footer
). Both are rendered from(site)/layout.tsx
. The fetch from the components gets revalidated and I can see the new data on all pages, except the/
route, aka.(site)/page.tsx
. That route never gets revalidated. Although the page and 2 components inside have tagged requests.In all 4 projects, the Sanity webhook calls the revalidate API route. The function always succeeds. Params are what they are supposed to be, so I guess the
revalidateTag()
gets called with the right parameters. (Although I have not figured out how to debug the actual function calls yet)Soooo I feel totally lost and I haven’t even tried to include previews and draft mode yet.
I'm not sure if it's a bug with Sanity, Next.js, or it's just me doing something wrong.
Verify canary release
Provide environment information
Which area(s) are affected? (Select all that apply)
App Router, Data fetching (gS(S)P, getInitialProps)
Additional context
Example fetch call (fails to revalidate, on / route):
Example fetch call (revalidates. /[slug] route):
sanityFetch function:
Revalidate route handler: