failing to render excalidraw image when filesize is too large

Hi @Mogztter and I do really apprechiate all you do for asciidoctor! :)

I've transferred the issue I've filed at kroki.

"I'm using excalidraw via asciidoc-kroki (from @Mogztter) - It seems that if the file gets to big or too many objects in the drawing it will fail to render the svg image. Attached both JSON files respective images of failing and the one working."

Success image sucess-image

Failure failing_excalidraw

The files that one is succeeding and the other is failing json-files.zip

Cheers, Mario :)

Just a quick follow-up question, you mentioned that one may increase the body size when self hosting - I've tried to set it to 1MB (since excalidraw seems to have 1m limit in it's server - https://github.com/yuzutech/kroki/blob/master/excalidraw/src/index.js#L14).

I've set the KROKI_BODY_LIMIT (https://github.com/yuzutech/kroki/blob/2bd2e664701f38877ba7ae75bfd833d0e1e5c0e7/server/src/main/java/io/kroki/server/Server.java#L67) environment variable in my docker compose - but it won't even log out that it tries to render it (failing json in above files). Is this incorrect way of increasing the body size?

my docker compose

version: "3"
services:
  core:
    image: yuzutech/kroki
    environment:
      - KROKI_BLOCKDIAG_HOST=blockdiag
      - KROKI_MERMAID_HOST=mermaid
      - KROKI_BPMN_HOST=bpmn
      - KROKI_EXCALIDRAW_HOST=excalidraw
      - KROKI_BODY_LIMIT=1048576
    ports:
      - "8000:8000"
  blockdiag:
    image: yuzutech/kroki-blockdiag
    expose:
      - "8001"
  mermaid:
    image: yuzutech/kroki-mermaid
    expose:
      - "8002"
  bpmn:
    image: yuzutech/kroki-bpmn
    expose:
      - "8003"
  excalidraw:
    image: yuzutech/kroki-excalidraw
    expose:
      - "8004"

Cheers, Mario :)

It's not the body size, it's the URI length. On https://kroki.io, we are using NGINX in front of the containers so the limit is enforced by NGINX (and the value is 4000). Vert.x also has a limit and as far as I know the default value is 4096. Currently, this value is not configurable using environment variable. For reference, we could update the value using https://vertx.io/docs/apidocs/io/vertx/core/http/HttpServerOptions.html#setMaxInitialLineLength-int-.

As a workaround you should use :kroki-fetch-diagram: to fetch the images using a POST request otherwise GET request will be denied.

As a reminder, if kroki-http-method is not defined, the extension will use a GET request if the URI length is <= 4096 and a POST request otherwise. You can use the :kroki-http-method: post to force the extension to send the diagram using a POST request and not a GET request.

My initial analysis was wrong since the URI length of the second diagram is greater than 4096. Having said that, for consistency, we should either update NGINX configuration to use 4096 or update the default value to 4000 in this extension: https://github.com/Mogztter/asciidoctor-kroki/blob/87a78b2c36ebac789a874a56d908c911f0269235/src/kroki-client.js#L26

Since you can potentially use a different value on a self-hosted instance, we might consider to make the value configurable (i.e., kroki-max-uri-length: 6000).

In summary, we should:

make the max URI length configurable on Vertx (using configuration)
make the max URI length configurable on this extension (using an AsciiDoc attributekroki-max-uri-length)
make the default value consistent across the ecosystem (either 4000 or 4096).

I will reopen your issue https://github.com/yuzutech/kroki/issues/555 and update the title to reflect what we plan to do.

@Mogztter Thanks for your help! I still do not understand how I'm about to tackle this issue. I'm always selfhosting kroki and sets the kroki options in both asciidoc header and in vscode settings (unsure where it picks them up from).

I do something totally wrong but I cannot figure out what!

Cheers, Mario :)

I've got the following asciidoc file:

(added both url len and switched to post just to get something to be rendered :)

ifndef::imagesdir[:imagesdir: ../../meta/assets]
ifndef::icons[:icons: font]
:kroki-http-method: post
:kroki-default-format: svg
:kroki-max-uri-length: 6000

= Overview of the Provider SDK

.Anatomy of a Provider
[excalidraw]
....
include::failing-excalidraw.json[]
....

in settings.json for the project:

"asciidoc.preview.attributes": {
  "kroki-http-method": "post",
  "kroki-max-uri-length":"6000",
  "kroki-server-url": "http://localhost:8000",
  "excalidraw-server-url": "http://localhost:8000",
  "plantuml-server-url": "http://localhost:8000"
},

The kroki server is dead silent (and preview in vscode is a broken image):

Creating network "kroki_default" with the default driver
Creating kroki_blockdiag_1  ... done
Creating kroki_mermaid_1    ... done
Creating kroki_excalidraw_1 ... done
Creating kroki_core_1       ... done
Creating kroki_bpmn_1       ... done
Hit Ctrl-C to stop kroki and dispose the containers
Attaching to kroki_core_1, kroki_bpmn_1, kroki_excalidraw_1, kroki_mermaid_1, kroki_blockdiag_1
blockdiag_1   | [2021-01-26 06:46:37 +0000] [1] [INFO] Starting gunicorn 20.0.4
blockdiag_1   | [2021-01-26 06:46:37 +0000] [1] [INFO] Listening at: http://0.0.0.0:8001 (1)
blockdiag_1   | [2021-01-26 06:46:37 +0000] [1] [INFO] Using worker: sync
blockdiag_1   | [2021-01-26 06:46:37 +0000] [9] [INFO] Booting worker with pid: 9
excalidraw_1  | Chrome accepting connections on endpoint ws://127.0.0.1:34579/devtools/browser/f6336149-892c-43b8-8376-b1f3fe964ddd
bpmn_1        | Chrome accepting connections on endpoint ws://127.0.0.1:34019/devtools/browser/40433f5c-33a8-438a-81d0-965ece1ef737
mermaid_1     | Chrome accepting connections on endpoint ws://127.0.0.1:34039/devtools/browser/924e054d-c31b-4449-a52f-013171a3e277
core_1        | Jan 26, 2021 6:46:39 AM io.vertx.core.impl.launcher.commands.VertxIsolatedDeployer
core_1        | INFO: Succeeded in deploying verticle

When I then hit ctrl-s to save the .vscode/settings.json it will output the following (I don't know if this is relevant):

core_1        | {"timestamp":"1611643925938","level":"INFO","thread":"vert.x-eventloop-thread-0","mdc":{"path":"/excalidraw/svg/eNoLzStKLc7PKUtNUUjJLEpNLsksS1XIzFOwKS5JycyzU9AFcpJzSlNSrazSEjNzMvPSdVMrkhNzMlOKEsv1sorz86JjAWe4GUo=","method":"GET","service_name":"excalidraw","action":"request_received","bytes_read":"0","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.52.1 Chrome/83.0.4103.122 Electron/9.3.5 Safari/537.36"},"logger":"io.kroki.server.service.DiagramHandler","message":"Request received GET /excalidraw/svg/eNoLzStKLc7PKUtNUUjJLEpNLsksS1XIzFOwKS5JycyzU9AFcpJzSlNSrazSEjNzMvPSdVMrkhNzMlOKEsv1sorz86JjAWe4GUo=","context":"default"}
core_1        | {"timestamp":"1611643925973","level":"INFO","thread":"vert.x-eventloop-thread-0","mdc":{"path":"/excalidraw/svg/eNoLzStKLc7PKUtNUUjJLEpNLsksS1XIzFOwKS5JycyzU9AFcpJzSlNSrazSEjNzMvPSdVMrkhNzMlOKEsv1sorz86JjAWe4GUo=","took":"32","method":"GET","service_name":"excalidraw","action":"convert","file_format":"svg","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.52.1 Chrome/83.0.4103.122 Electron/9.3.5 Safari/537.36"},"logger":"io.kroki.server.service.DiagramHandler","message":"Convert took 32ms","context":"default"}
excalidraw_1  | Unable to convert the diagram Error: Evaluation failed: SyntaxError: Unexpected token U in JSON at position 0
excalidraw_1  |     at JSON.parse (<anonymous>)
excalidraw_1  |     at __puppeteer_evaluation_script__:2:68
excalidraw_1  |     at ExecutionContext._evaluateInternal (/snapshot/app/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:217:19)
excalidraw_1  |     at process._tickCallback (internal/process/next_tick.js:68:7)
excalidraw_1  | e Error: Evaluation failed: SyntaxError: Unexpected token U in JSON at position 0
excalidraw_1  |     at JSON.parse (<anonymous>)
excalidraw_1  |     at __puppeteer_evaluation_script__:2:68
excalidraw_1  |     at ExecutionContext._evaluateInternal (/snapshot/app/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:217:19)
excalidraw_1  |     at process._tickCallback (internal/process/next_tick.js:68:7)
core_1        | {"timestamp":"1611643926130","level":"ERROR","thread":"vert.x-eventloop-thread-0","mdc":{"error_message":"Unable to convert the diagram","path":"excalidraw:8004/excalidraw/svg","method":"POST","action":"delegate","error_code":"400"},"logger":"io.kroki.server.action.Delegator","message":"Unsuccessful request POST excalidraw:8004/excalidraw/svg - response: {statusCode:400 body: Unable to convert the diagram}","context":"default"}
core_1        | {"timestamp":"1611643926131","level":"ERROR","thread":"vert.x-eventloop-thread-0","mdc":{"error_message":"OK","action":"error","path":"/excalidraw/svg/eNoLzStKLc7PKUtNUUjJLEpNLsksS1XIzFOwKS5JycyzU9AFcpJzSlNSrazSEjNzMvPSdVMrkhNzMlOKEsv1sorz86JjAWe4GUo=","error_code":"400","method":"GET","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.52.1 Chrome/83.0.4103.122 Electron/9.3.5 Safari/537.36"},"logger":"io.kroki.server.error.ErrorHandler","message":"An error occurred","context":"default"}

I think you forgot to use :kroki-fetch-diagram:

As a workaround you should use :kroki-fetch-diagram: to fetch the images using a POST request otherwise GET request will be denied.

Without this option, Kroki will convert the excalidraw block to an <img> element:

<img src="http://localhost:8000/excalidraw/svg/...."/>

Then, the browser (or in your case the VS Code preview pane) will render the HTML document and use a GET request to fetch the image!

When using :kroki-fetch-diagram:, the extension will fetch the image at "conversation-time" (i.e., before the HTML document is generated). In other words, the extension will fetch the image from the remote server (http://localhost:8000) using either a GET or POST request and write the image to the disk. The extension will still convert the excalidraw block to an <img> element but using a local path:

<img src="diagram.svg"/>

Does it make sense?

Thanks again @marcelstoer ! This works as a charm! And I'm finally get it... - Albeit a bit slow ;)

Cheers, Mario :)

asciidoctor / asciidoctor-kroki

failing to render excalidraw image when filesize is too large #194