Form integrations with large payloads post_max_size and input_max_vars on Craft Cloud

[jamesmacwhite]

Describe the bug

Formie forms using integrations with large field mappings or where the payload is significant due to the API integration setup, has reared its head again, this time on Craft Cloud.

If you remember the fun debugging to find out why forms were losing changes or deleting fields randomly here: https://github.com/verbb/formie/issues/1212#issuecomment-1362696377

The issue I believe is due to the fact post_max_size is set to 6M and the input_max_vars is 1000. I have opened a discussion on the Craft Cloud repo and and sent a support email about the issue if there's anything that can be done to amend this.

We are running Formie v3 on a project, but experiencing the same issues, but this time I know why, due to the PHP env vars. Craft Cloud is trickier because of its serverless nature, meaning no direct access to change the PHP environment. I'm hoping the Craft Cloud team are able to implement changes, because it is unfortunately breaking Formie forms and could break others too.

However, I don't know if anything has been done regarding integrations and payloads since the original issue, the original issue was on Formie V1 for Craft CMS 3.

https://github.com/craftcms/cloud/discussions/45

From the plugin side, I don't know if there's any handling that can be done to help remedy the issue. I feel a post_max_size of 6M is too low generally, although it may be input_max_vars that is actually the main problem based on what I found previously.

While Formie isn't officially Craft Cloud certified under the additional verification program, it does seem like a potential issue others could encounter, but mostly dependant on the form integrations configured.

Steps to reproduce

1. Deploy a Craft CMS site on Craft Cloud
2. Install Formie
3. Setup an integration which has lots of fields/mapping areas e.g. Microsoft Dynamics 365
4. Create a form and test saving changes to properties
5. Observe changes being lost despite saving in the Control Panel.

Form settings

N/A

Craft CMS version

5.4.9

Plugin version

3.0.9

Multi-site?

No

Additional context

No response

[engram-design]

I'm not exactly sure what I can do about that, short of splitting the payload for forms into multiple parts. I'm almost certain this is going to be an issue for other plugins, particular Super Table, or ever just last entries.

I suppose I could exclude any integration field that you don't map to not be sent to the server.

I'm not sure how complex your forms are, but I would see the exact same thing happening for a form with a large amount of fields. I really don't want to go down the path that Craft did with delta updates (only sending fields that have changed), which just adds complexity to the plugin and more failure points.

I'd love to see Craft fix this on their end!

[jamesmacwhite]

@engram-design I'm speaking with @angrybrad about this currently through Craft Cloud support.

Doing some testing locally on DDEV the main PHP var appears to be input_max_vars, rather than post_max_size. The issue with the latter is Craft Cloud is limited by AWS Lambda synchronous limitations of 6MB. However, the initial area of concern is the input_max_vars setting being 1000, bumping to 5000 seems to work OK. I have ran DDEV locally with the 6M post_max_size and the bumped input_max_vars setting and was able to work with Formie forms OK with the Microsoft Dynamics 365 CRM integration configured.

I am still concerned about just having just 6M because it does seem like it could cause problems in other contexts, but that's more of a hard limitation set by the AWS infrastructure behind Craft Cloud from what I understand, rather than an opinionated decision.

On the Formie side, I do think there are two areas to help with request sizes generally on forms using integrations.

1. If an integration is disabled, the payload shouldn't include the field mapping, given it is not active. Looking through logs, it did appear this was happening, but I could be wrong.
2. If an integration is enabled any empty field that is not mapped to a value i.e. '' could be stripped out from the payload, that would significantly reduce payloads currently.

This would likely have more benefit to CRM integrations, given there typically larger on average.

Using Microsoft Dynamics 365 CRM as the example, you have several entities, leads, contacts, opportunity, organisations. Under each of these entities are many fields, that's a lot of empty fields to have sent, for when you've probably only mapped a handful. While the payload is an associative array of integration field handles and string values, that's still a lot to send potentially as empty.

I certainly don't expect Formie to workaround potential architecture/AWS limitations, but I think the two areas above would have benefit for the plugin as a whole, while likely reducing the risk of hitting that post_max_size value on environments where it is perhaps more constrained. I don't know if there's any official specific Craft CMS optimised PHP ini values beyond having certain PHP functions available.

I'm somewhat hoping that an integration payload despite having a large amount of entities with multiple associative arrays, because it is just field handles and string values, that it can stay within that 6M limit, because the data is relatively small and the input_max_vars is more of the issue really.

I don't know how many large scale Craft CMS sites are hosted on Craft Cloud currently, but I am surprised this hasn't caused issues for someone already, as I imagine Formie is quite popular. I guess it is only apparent with certain form integrations which are larger. We only found the issue when we starting using form integrations and maybe that is the key factor. It is both lucky and unlucky that I knew immediately what the issue was, having previously gone through the environmental debugging with you previously!

[engram-design]

Agreed on both fronts that integrations can be optimised like that, but as I mentioned, any sufficiently large form is going to hit the same issue pretty quickly, so it'll just be fixing one problem for it to happen elsewhere - not really fixing the core issue.

[jamesmacwhite]

@engram-design I understand what you are saying. I really do. The main points where that the payload size of form integrations is possibly bloated so it could be improved generally. Having a value of 6M is a completely different issue which isn't really a Formie issue, I was just giving the context as to the limitation working in.

Unfortunately, post_max_size is something tied to Craft Cloud's serverless infrastructure and is more of a hard limit. The source of this is AWS Lambda on synchronous requests.

The 6MB limit on requests:

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

From the Craft Cloud knowledgebase:

https://craftcms.com/knowledge-base/cloud-quotas#requests-responses

While we don’t impose limits on the number of total or concurrent requests, it’s important that they each finish within 28 seconds. Responses with headers exceeding 16,000 bytes (total) may be dropped, so avoid sending long identifiers or setting large cookie values. If you need to associate a significant amount of data with a visitor, consider using the cache or a database table and tying it to the session by ID. Additionally, the maximum total response length is 6MB, before compression (regardless of its Content-Type). The average HTML document size is [about 30KB](https://almanac.httparchive.org/en/2022/markup#document-size), or roughly 0.5% of our cap—so very few sites and apps should be impacted. Asset uploads from the Craft control panel are not subject to this 6MB limit, as they are uploaded directly to your bucket and served from our CDN. Asset uploads from the site’s front end are subject to this limit unless you provide a way to upload them directly to your bucket from the front end (similar to how the Craft control panel does it). Build artifacts are also not subject to this limit, as they are served from our CDN.

There's absolutely nothing that can be done as it is a Craft Cloud infrastructure limitation by the sounds of it. That's what I've been told by @angrybrad and it is stated in the Craft Cloud docs.

Ultimately, Craft Cloud may not be for us based on the above which is a shame, given it is a first-party solution, but there are other SaaS providers out there.

[engram-design]

Just to be clear, I'll be doing those integration updates like you mentioned, but just that it might be an issue that crops up again for another reason (large form with lots of fields), but hopefully addressing this for integrations will fix this one up for you right now!

[jamesmacwhite]

Thank you, it is appreciated, I feel that is a good thing to help with payload sizes generally.

The 6M limit is something I need to balance generally. I don't disagree that other non-formie scenarios could cause issues around it.

[engram-design]

Fixed for the next release. To get this early, run composer require verbb/formie:"dev-craft-5 as 3.0.9".

[jamesmacwhite]

Thanks @engram-design! For further info on this, Craft Cloud has a undocumented way of changing PHP configuration ini vars, so I was able to set input_max_vars to 5000 to implement a fix. These changes will help with payload sizes generally though for sure.

The post_max_size cannot be changed due to using AWS Lambda for serverless on Craft Cloud as mentioned, the Craft team think that other than front-end uploads, having 6 MB of purely textual POST data would be an unusual scenario.

For front-end uploads there is a solution to bypass Lambda and instead send data to S3 (not limited by the 6MB limit), but it would require a slightly different approach. In the context of Formie, the file upload/front end uploading would be the area. The Craft team have said they plan to provide documentation on being able to implement the Craft Cloud S3 upload method, similar to how to the Craft Cloud filesystem type works, that allows for larger uploads without restrictions.

I believe developers are being offered sandboxes for testing on Craft Cloud, if it is something you are interested in or if you wanted to have Formie verified on Craft Cloud through the additional verification program, that is possible:

https://craftcms.com/knowledge-base/cloud-plugin-development

Happy to also provide feedback, as we will be running a production site on Craft Cloud soon.

[engram-design]

Fixed in 3.0.10