search

dfpc-coe / CloudTAK

Full Featured in-browser TAK Client powered by AWS
GNU Affero General Public License v3.0
14 stars 2 forks source link

CoT Payload Too Large Error #69

Closed ingalls closed 3 months ago

ingalls commented 1 year ago

Context

The NIFC layer is occasionally throwing the following error:

2023-05-18T19:58:10.865Z    d027eae3-d9f9-4bf3-8925-82033d688e9d    ERROR   <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>PayloadTooLargeError: request entity too large<br> &nbsp; &nbsp;at readStream (/home/etl/api/node_modules/raw-body/index.js:163:17)<br> &nbsp; &nbsp;at getRawBody (/home/etl/api/node_modules/raw-body/index.js:116:12)<br> &nbsp; &nbsp;at read (/home/etl/api/node_modules/body-parser/lib/read.js:79:3)<br> &nbsp; &nbsp;at rawParser (/home/etl/api/node_modules/body-parser/lib/types/raw.js:81:5)<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/etl/api/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at next (/home/etl/api/node_modules/express/lib/router/route.js:144:13)<br> &nbsp; &nbsp;at file:///home/etl/api/node_modules/@openaddresses/batch-schema/lib/middleware.js:126:13<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/etl/api/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at next (/home/etl/api/node_modules/express/lib/router/route.js:144:13)<br> &nbsp; &nbsp;at /home/etl/api/node_modules/express-json-validator-middleware/src/index.js:63:5<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/etl/api/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at next (/home/etl/api/node_modules/express/lib/router/route.js:144:13)<br> &nbsp; &nbsp;at file:///home/etl/api/node_modules/@openaddresses/batch-schema/lib/middleware.js:35:20<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/etl/api/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at next (/home/etl/api/node_modules/express/lib/router/route.js:144:13)<br> &nbsp; &nbsp;at file:///home/etl/api/node_modules/@openaddresses/batch-schema/lib/middleware.js:98:20</pre>
</body>
</html>

depending on how much data it is handling.

ingalls commented 1 year ago

This issue has been partially resolved for CoT submissions to a live endpoint. Submissions using the etl-base library will be automatically broken up into smaller sections which are below this limit.

However submissions to the data endpoint that exceed this limit and are broken into multiple CoT submissions will result in them overwriting the data submission on S3.