Response payload size exceeded maximum allowed payload size

[tdelmas]

Validation with https://api-public.odpt.org/api/v4/gbfs/hellocycling/gbfs.json :

{ "errorMessage": "Response payload size exceeded maximum allowed payload size (6291556 bytes).", "errorType": "Function.ResponseSizeTooLarge" }

[PierrickP]

Probably nothing to do :\ https://answers.netlify.com/t/lambda-functions-response-size-too-large/22374/2

[davidgamez]

As an alternative solution, we can follow the GTFS validator's pattern, create the JSON and HTML reports as separate files and make them available for download. The HTML version can be rendered in the bottom section of the current UI; this will require more work and UI layout changes. Any thoughts?

[davidgamez]

Another solution is using streaming to return the response, but I did some investigation and found that streaming is not supported yet, by Netlify functions, even though the under-the-hood AWS lambda functions already do. More info here, https://answers.netlify.com/t/stream-content-to-function-response-body/3558/20.

[richfab]

Another feed example with this error: https://api.entur.io/mobility/v2/gbfs/v3beta/voi/gbfs – found in https://api.entur.io/mobility/v2/gbfs/v3beta/manifest.json.

[richfab]

Potential improvement to reduce response payload size (discussed with @PierrickP): stop sending the actual GBFS feed data in the validation response to the client.

For the visualization part, we would still need to pass some GBFS feed data to the client.

[derhuerst]

Would a CE-coded (e.g. gzipped or brotli-compressed) response stay beneath the limit? Highly repetitive text like validation reports compress especially well, likely to <10% of the original size.

For example, if the client requests with Accept-Encoding: gzip, the server can respond with Content-Encoding: gzip and a gzipped body.

[PierrickP]

@derhuerst Sadly, it already uses Brotli

[derhuerst]

Sadly, it already uses Brotli

Is that the Netlify function, or is it the HTTP reverse proxy in front of it?

[PierrickP]

Directly handled by Netlify. Do you think we can send more if we compress before ?

[derhuerst]

It's worth a try I'd say!