/api/access response exceeds 4MB Vercel limit

[rafalnawojczyk]

Link to reproduction

No response

Describe the Bug

We have recently established an extensive design system centered around PayloadCMS, employing a combination of Serverless MongoDB, Serverless Vercel, and Serverless PayloadCMS. However, we have encountered a significant issue post-successful login, wherein Payload initiates a request to the "/api/access" endpoint to retrieve all permissions. The challenge lies in the fact that it creates permissions for every element and every field, resulting in an exponential growth of data.

To provide a contextual overview of our project structure, which is built on Next.js, we've organized components into three key categories:

Sections: These are equivalent to Payload blocks or modules. UI Elements: Smaller components nested within Sections to enhance flexibility. Utils: Atom components serving various utility functions.

This modular approach allows us to create diverse compositions, such as a Hero Section with an image on the right side and customizable UI elements on the left side, ranging from buttons to lists.

Our project in numbers:

114 Sections 43 UI Elements 20 Utils

The challenge arises during the generation of the permissions file associated with the response from /api/access. The file includes redundant code for each collection, leading to an extensive repetition that results in a JSON file exceeding Vercel's 4.5MB limit. The endpoint response, therefore, consists of a JSON file with over 436,000 lines.

Here comes the question - is there an option to disable this type of permission check, and change it to key based only with predefined roles like 'admin', 'editor' where we could specify their permissions? I think that right now this system generates way to much code.

The same problem can be seen in payload-types.ts file, where redundancy is on the same level. Even with predefined fields that are the same in every Block - it is creating them from scratch each time they are used within section.

Is there an easy/fast fix to disable these permissions so we would work until our MVP deadline with reduced security?

To Reproduce

Basically problem arises itself when there is more sections/blocks.

Payload Version

2.1.1

Adapters and Plugins

dn-mongodb

[robertbabinski]

BUMP 👀

Searching for the same answer for the issue provided by @rafalnawojczyk.

One important thing - we are using the next-payload package.

[rafalnawojczyk]

Maybe I will add one possible solution for that. Maybe you should consider adding some "factory fields" that are immutable, so when developers will use this factory to create some fields - it grants that they are immutable, and they will never changes. So this makes it easier to reuse types/access permissions across the app. We are making generic fields like one for Icon/Button that we are reusing very commonly throughout our app, as thats the main part of React - reusability of smaller components.

Adding something like immutable fields generators will have huge impact on /api/access response size(depends on implementation) and/or on payload-types.ts size.

Another thing is maybe changing whole permission access by some fields in config. Like right now it is kinda like whitelisted - so user has no access to any field, and per-field he has permissions granted. This flag in config could replace it with blacklisted, so user have access to all fields(except fields that he doesnt have permissions to). In our case, the second solution(blacklisted pattern) would reduce the '/api/access' size to literally zero, as in our project, in 500k lines of access response - there is almost no single 'false'.

[DanRibbens]

@rafalnawojczyk,

I like this line of thinking, but the behavior of true vs false is entirely subjective to your access control and authenticating user.

On the immutability "factory fields" idea, we don't want to burden the developers writing the config as that becomes especially complicated for plugins and projects integrating well together. I'm not sure if you're thinking about something that is config driven or is happening internal to Payload.

My initial thoughts on a proposed solution is that the access api could return a much smaller response if we used a little more logic to truncate to true or false at the highest possible key. Both the frontend and backend would need to be changed, but this way the only time the entire tree that we currently return would come through is when you have a mix of true/false down to the field access level.

As an aside, instead of “whitelist” and “blacklist”, we can use “allowlist” and “denylist”.

[DanRibbens]

Can somebody add me to a repo or link to one where this problem is occuring? I want to understand better about this problem before I go deep on implementing a solution.

[robertbabinski]

Hey @DanRibbens - unfortunately we can't add you to the client project where the issue appears. 😢

We will try to reproduce this issue in the repo next to our project, and then invite you. 😄

[JarrodMFlesch]

The same problem can be seen in payload-types.ts file, where redundancy is on the same level. Even with predefined fields that are the same in every Block - it is creating them from scratch each time they are used within section.

I think this could be solved by implementing interfaceNames for fields that get used a lot. Check out this section of our docs to read a bit about them!

[jmikrut]

Yes I agree with @JarrodMFlesch — using interfaceName can solve this for the generated types aspect. It would also help with GraphQL.

Outside of that, I am going to close this issue as I really think that there are much bigger problems awaiting you if the /api/access endpoint returns a response larger than 4MB.

We could optimize that single endpoint, but I've got to imagine that starting up Payload is getting a bit slow? Especially if you're running on Vercel serverless, you've got to be seeing a lot of cold start delays. Payload would have to do a LOT of work up front to start up if you truly have 4mb+ of response size for your access operation.

This just seems to be something that might need to be solved via architecting your schema differently. I would look into using relationships more heavily.

Have any of you made it to production with this issue?

I would like to learn more about the other side-effects of using Payload in this manner.

• How many fields do you end up having per collection, total?
• Is the GraphQL endpoint slow? What does your GraphQL schema end up looking like? Horrible?
• What is the startup time like for Payload?
• How bad are cold starts, for those of you deploying on Vercel?
• Does the admin panel "lag" at all?

as in our project, in 500k lines of access response - there is almost no single 'false'.

For cases like this, we can certainly truncate branches of the access response that all return either true or false. But as I said above, that seems to be putting a band-aid over a larger issue. I will open that as a feature request if we can get some more certainty that Payload otherwise handles this many fields properly. It would be surprising to me if it did.

[rafalnawojczyk]

Payload CMS is working pretty fast. Only when you are trying to make a page that has 50+ blocks - then it starts to slow down really fast. Up until one operation(adding one letter in slate rich text) takes around 3-4 seconds to show up in UI.

We are working on splitting PayloadCMS from our website project, as it producing more problems than just /api/access beeing to big.

Right now our main performance issue is that if page is getting any data from PayloadCMS - it is adding a lot of unused JS to the generated page(around 1MB).

After this step(splitting codebases) - it will be easier for us to give you example repo, because our Payload repo will be free from any classified content. It is going to be done(I hope) this month, so I will just reply in this thread once again with all the informations.

On top of that, when example repo will be ready - I will write down more issues and link them together, because all of them can be tested on that repo.

[KnotScientific]

Hi @rafalnawojczyk, any updates on this issue and any potential steps you did to reduce the slowdown? I have also recently started working on a 50+ blocks, nested website and am feeling the CMS slowing down.

[rafalnawojczyk]

Hi @rafalnawojczyk, any updates on this issue and any potential steps you did to reduce the slowdown? I have also recently started working on a 50+ blocks, nested website and am feeling the CMS slowing down.

Hi, we've been trying to fix this issue on our own, and unfortunately - we cannot make any improvements in terms of speed on pages that are using lots of blocks. The only one "hotfix" that you can apply is to Collapse all the blocks that are currently inside CMS, refresh page and start working again. When fields are collapsed - everything works way faster.

I was thinking about showing example repo, but right now I'm waiting for Payload 3.0, and hopefully it will resolve this issue. If not - I will definitely make a contribution or prepare example repo.

[KnotScientific]

I see, that's a bummer. I'm curious as to how long your yarn dev takes to coldstart the website and admin UI. For me 50+ blocks it's taking me 13min. I feel like it shouldn't take that long and I'm missing something. The loadconfig step in payload.init generates the JSON config file of my whole website and it seems to be the bottleneck. Wondering if you were experiencing the same.

[rafalnawojczyk]

@KnotScientific Are you generating types for payload in your dev command? This can slow down the process, and it's not necessary to rebuild types, as long as nothing changed in your definitions.

For us it takes around 5-9s for Next.js to start, and then after trying to access Admin UI - it takes 60-80s to pregenerate and build all deps.

Our whole build for 2k+ pages takes around 17mins.

If your project is open on github - I could take a look and maybe guide you, but it's outside of this issue I think

[KnotScientific]

@KnotScientific Are you generating types for payload in your dev command? This can slow down the process, and it's not necessary to rebuild types, as long as nothing changed in your definitions.

For us it takes around 5-9s for Next.js to start, and then after trying to access Admin UI - it takes 60-80s to pregenerate and build all deps.

Our whole build for 2k+ pages takes around 17mins.

If your project is open on github - I could take a look and maybe guide you, but it's outside of this issue I think

Hmm, I'm not sure. I think it is because the performance is similar to the generate:types command. Adding some loggers into the payload.init function, the loadconfig function from line 202 in the pic below compiles the config JSON and that's what takes the longest time, so maybe it's generating the types? Not sure how I would disable that since it's part of the init function.

I haven't touched anything on the config side of things. I'm using their nextjs ecommerce template, all commands are untouched from this repo. https://github.com/payloadcms/payload/tree/main/templates/ecommerce

[github-actions[bot]]

This issue has been automatically locked. Please open a new issue if this issue persists with any additional detail.