search

sagemathinc / cocalc

CoCalc: Collaborative Calculation in the Cloud
https://CoCalc.com
Other
1.16k stars 211 forks source link

write_text_file_to_project triggers PayloadTooLargeError for large content #4034

Open haraldschilly opened 5 years ago

haraldschilly commented 5 years ago

this does happen in particular when converting a large sagews file to html. in the last step, the content is written to a file via the client call: write_text_file_to_project. In a case where it fails, the content's length is 2866497.

This is the error: 

PayloadTooLargeError: request entity too large
    at readStream (/home/user/cocalc/src/smc-hub/node_modules/raw-body/index.js:118:13)
    at getRawBody (/home/user/cocalc/src/smc-hub/node_modules/raw-body/index.js:92:13)
    at read (/home/user/cocalc/src/smc-hub/node_modules/body-parser/lib/read.js:77:3)
    at urlencodedParser (/home/user/cocalc/src/smc-hub/node_modules/body-parser/lib/types/urlencoded.js:116:5)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at jsonParser (/home/user/cocalc/src/smc-hub/node_modules/body-parser/lib/types/json.js:89:14)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at Function.handle (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:174:3)
    at router (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:62:16)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at compression (/home/user/cocalc/src/smc-hub/node_modules/compression/index.js:220:5)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at cookieParser (/home/user/cocalc/src/smc-hub/node_modules/cookie-parser/index.js:71:5)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at expressInit (/home/user/cocalc/src/smc-hub/node_modules/express/lib/middleware/init.js:57:9)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at query (/home/user/cocalc/src/smc-hub/node_modules/express/lib/middleware/query.js:61:9)
    at Layer.handle [as handle_request] (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at /home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13
    at Function.process_params (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at next (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:182:13)
    at Function.handle (/home/user/cocalc/src/smc-hub/node_modules/express/lib/router/index.js:174:3)
    at Function.handle (/home/user/cocalc/src/smc-hub/node_modules/express/lib/application.js:174:10)
    at Server.app (/home/user/cocalc/src/smc-hub/node_modules/express/lib/express.js:56:11)
    at child.emit (/home/user/cocalc/src/smc-hub/node_modules/eventemitter3/index.js:141:17)
    at child.request (/home/user/cocalc/src/smc-hub/node_modules/primus/transformer.js:118:13)
    at Server.emit (events.js:153:21)
    at parserOnIncoming (_http_server.js:182:13)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:93:16)
williamstein commented 5 years ago

Instead of the old way of routing this through a hub, we should instead either use the project raw server directly somehow (?), or the project websocket. It would only take a few minutes to add this to the project websocket's api, but I don't know whether or not it would hit any size limit issues.

williamstein commented 5 years ago

converting a large sagews file to html.

For that problem, the other approach would be to move this to the backend, if possible. I.e, make it code that just runs 100% in the project.