Converting HTML to project causing Entity Too Large error

[samtlewis]

On certain templates, when we try to use the api to convert to a project, we'll get an 413 Entity Too Large error from the EDM side. It looks like the html is converted to the document object but when we pass that to the "/project/create" path that is where we are seeing the Entity Too Large error. This is not with every template we import but seems to be with some more complex templates.

Is there a restriction on the size of a template that can be created using the "/project/create" method?

We are also seeing intermittent 504 Gateway timeout issues on api-import.edmdesigner.com as well.

`* Hostname in DNS cache was stale, zapped

• Trying 35.169.222.138...
• TCP_NODELAY set
• Connected to api-import.edmdesigner.com (35.169.222.138) port 443 (#0)
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
• successfully set certificate verify locations:
• CAfile: /Applications/MAMP/Library/OpenSSL/cert.pem CApath: none
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server accepted to use http/1.1
• Server certificate:
• subject: OU=Domain Control Validated; CN=*.edmdesigner.com
• start date: Feb 26 18:41:02 2018 GMT
• expire date: Apr 27 09:07:38 2019 GMT
• subjectAltName: host "api-import.edmdesigner.com" matched cert's "*.edmdesigner.com"
• issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
• SSL certificate verify ok.

  POST /import?token=XXXXXXXXX HTTP/1.1 Host: api-import.edmdesigner.com Accept: / Content-Length: 89679 Content-Type: application/x-www-form-urlencoded Expect: 100-continue

< HTTP/1.1 100 Continue

• We are completely uploaded and fine < HTTP/1.1 504 Gateway Time-out < Server: nginx/1.10.3 (Ubuntu) < Date: Thu, 07 Jun 2018 14:25:07 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Cache-Control: no-cache <
• Curl_http_done: called premature == 0
• Connection #0 to host api-import.edmdesigner.com left intact

[smiska]

Hi Sam,

There's a restriction of 2 MB per request body size in our server. Do you think it's possible that your project could exceed it? Could you send a projectID so I can check?

I added a colleague to this thread. I hope he has more information on this. Do you experience this in specific scenarios or is it a generally occurring issue?

[samtlewis]

The template we converted was not large at all. It was an 89kb file. But the json document was huge when it was returned. I think it came in at 5.8mb. Not sure I understand the increase in file size by that extent. I can't provide a projectID since the project creation fails. I can send the html that is converted if you like. What email should I send it to?

Just to confirm the process: We use the import html api to convert the html to the json object (89kb file uploaded, 5.8mb json object returned) We take the json object returned and try to create a project from it. The create project api call fails with the entity too large error.

[smiska]

It seems that the process fails because the JSON is too large for the API call.

Could you send the HTML and JSON to support@edmdesigner.com ? We will investigate this. I don't have in-depth knowledge about the import tool, but this JSON size is surprisingly large.

We could elevate the request body restriction, but I believe that it may be a better approach to make adjustments to the import tool.

[samtlewis]

@smiska I just emailed the html.

[smiska]

@samtlewis I notified @stubendek about it, we received the HTML thanks. We'll investigate it. There are special characters in the HTML, that could have an effect on the resulting size, but we'll take a deeper look at it.

[samtlewis]

@stubendek @smiska - Any update on this? We've got a number of clients being affected currently and it's holding them up. Is this an issue on the html import or something we may be doing on our side?

[stubendek]

@samtlewis , we are working on the issue. we localized the error, it affects the text recongition core, it takes time to correct it. however, there are some tricks how to avoid the error: 1) the most important thing is to wrap texts in a paragraph tag (P), that corrects the large json issue and the aligning as well 2) The html tag BIG is deprecated, thus it is removed from the source, try using styles. 3) Try avoiding non-html characters, like the \95 We will send a corrected version of the mail.

[samtlewis]

Great, thanks for the update. We don't have control of the html but we'll advise our clients of these steps.

[samtlewis]

Any update on this issue? Any changes made to the conversion process to reduce the file size and avoid these errors?