Edge/Cloudflare Deployments fail due to `api/_content/cache.123.json` size

warflash commented 10 months ago

Environment

Operating System: Windows_NT
Node Version: v18.18.2
Nuxt Version: 3.8.1
CLI Version: 3.9.1
Nitro Version: 2.7.2
Package Manager: npm@8.13.2
Builder: -
User Config: runtimeConfig, build, app, nitro, modules, tailwindcss, css, schemaOrg, vite, typescript, alias, experimental, ogImage
Runtime Modules: @pinia/nuxt@0.4.11, @nuxtjs/tailwindcss@6.8.0, nuxt-headlessui@1.1.4, nuxt-icon@0.3.3, @nuxt/content@2.9.0
Build Modules: -

Reproduction

-

Describe the bug

Cloudflare pages and I assume other edge products based on cf workers and KV are limited to 25mb in single file size. The cache fill nuxt content generates may easily surpass that. ->

✘ [ERROR] Error: Pages only supports files up to 26.2 MB in size
api/_content/cache.1699317964624.json is 47.5 MB in size

Additional context

No response

Logs

1:51:57.934 | Found _routes.json in output directory. Uploading.
01:51:57.954 | Validating asset output directory
01:51:59.198 | Deploying your site to Cloudflare's global network...
01:52:06.292 |  
01:52:06.424 | ✘ [ERROR] Error: Pages only supports files up to 26.2 MB in size
01:52:06.425 |  
01:52:06.425 | api/_content/cache.1699317964624.json is 47.5 MB in size

Barbapapazes commented 10 months ago

Hello,

47mb is huge. what is the content of this file?

warflash commented 10 months ago

As far as I can tell it contains the parsed AST of all files in the /content dir. Our content directory hold about 26mb of content which when built turns into ~46mb of AST cache. This is the cache file of the starter template after building the project, adding more content will make that grow: cache.1699359927415.json

I general I feel like it might not be the best solution performance wise to load and parse such a huge json object on every content requests. On my local machine reading and parsing our cache file takes ~400ms alone so a single lamdba might struggle even more when loading that

Barbapapazes commented 10 months ago

Yes, I see. this endpoint is only used by the clientDb. maybe it could be disabled using an option (not implemented)? (but this could impact nuxt studio, don't know more about this)

warflash commented 10 months ago

Interesting.. Yeah I guess if it's really not needed then an option to disable generation might be handy indeed. According to the logs it takes about 80s to generate the cache file for us as well so that'd be quite the savings in the CI itself. There is content.experimental.clientdb but that doesn't seem to do the trick

farnabaz commented 9 months ago

For now, you can set experimental.cacheContents to false in order to disable this chunk. The fix is released in edge-channel, You can try it in edge version.

nuxt / content