spawnSync /bin/sh ENOBUFS is thrown when the POST data is greater than 1 megabytes

[mariotoffia]

Hi again and many thanks for this great software! :)

I've stumbled into a problem again. I'm unsure if it is kroki or asciidoc-kroki. I'm guessing, there's something that I do something incorrect - again...

Cheers, Mario :)

I've got the following settings:

"asciidoc.preview.attributes": {
  "kroki-http-method": "post",
  "kroki-max-uri-length":"6000",
  "kroki-fetch-diagram": "true",
  "kroki-server-url": "http://localhost:8000",
  "excalidraw-server-url": "http://localhost:8000",
  "plantuml-server-url": "http://localhost:8000"
}

.Working
[excalidraw]
....
include::storage-encapsulation.json[]
....

.Not working
[excalidraw]
....
include::sample-pipe.json[]
....

I've attached a zip file with a working excalidraw (storage-encapsulation.json) and the failing one (sample-pipe.json). excalidraw.zip

The log I get from kroki when doing the failing is the following

core_1        | {"timestamp":"1611728581739","level":"INFO","thread":"vert.x-eventloop-thread-0","mdc":{"path":"/excalidraw/svg","method":"POST","service_name":"excalidraw","action":"request_received","bytes_read":"41809","user_agent":"node-XMLHttpRequest"},"logger":"io.kroki.server.service.DiagramHandler","message":"Request received POST /excalidraw/svg","context":"default"}
core_1        | {"timestamp":"1611728581739","level":"INFO","thread":"vert.x-eventloop-thread-0","mdc":{"path":"/excalidraw/svg","took":"0","method":"POST","service_name":"excalidraw","action":"convert","file_format":"svg","user_agent":"node-XMLHttpRequest"},"logger":"io.kroki.server.service.DiagramHandler","message":"Convert took 0ms","context":"default"}

No svg file is written as meta/assets/diag-uuid.svg.

I'm expecting the following output in the asciidoc preview

But I get the JSON instead (if it's not working, it usually outputs a broken image)

[ggrossetie]

Hey @mariotoffia

Thanks for sharing your diagrams it helps to quickly reproduce issues. Here's what I get:

$ asciidoctor -r asciidoctor-kroki doc.adoc 
Skipping excalidraw block. POST https://kroki.io/excalidraw/svg - error; reason: spawnSync /bin/sh ENOBUFS

The second diagram is exceeding the max buffer size of the child_process.execSync function.

If you are using Node 12 or greater the default value for max buffer is 1024 1024 bytes (~1 megabytes). If you are using an older version of Node then the value is 200 1024 bytes (~200 kilobytes).

This function is used in the unxhr library to make synchronous HTTP calls.

As far as I know, it's not possible to configure Node to increase this value. The only way to change this value is to explicitly define the maxBuffer option on the child_process.execSync function in here:

https://github.com/Mogztter/unxhr/blob/87da78d926ad5161223861ec40be74bcb66e4dec/lib/XMLHttpRequest.js#L514-L517

For reference, a "popular" library is using 100 MB as the default value: https://github.com/sindresorhus/execa/blob/02a8fe8459e5e86e5c8eb9c1c2d92e5b93901355/index.js#L15

So it's probably safe to increase this value. Anyway I don't think this value should be enforced by this option but rather by the memory available/allocated to the Node process. If you are working on large data (and you are allocating the right amount of memory to your Node process) then the default value of 1 megabytes is arbitrary and does not make much sense.

I will transfer this issue to the unxhr project :arrow_right:

[mariotoffia]

@Mogztter again many thanks for your great support! :) I'm using the the docker-compose.xml from kroki.io and I'm guessing it uses this where it declares

# Package the Node.js project into a single binary
FROM node:12.18.4-alpine3.11 as builder

I will track the progress in the unxhr!

Cheers, Mario :)

[ggrossetie]

again many thanks for your great support! :) I'm using the the docker-compose.xml from kroki.io and I'm guessing it uses this where it declares

I was asking about the version of Node used client-side (by VS Code) to execute the Asciidoctor Kroki extension. unxhr is used to send requests from the client to the server.

But in any case the limit is too low :wink:

[mariotoffia]

Hahahaha sorry about that! :D - vscode is running node v12.14.1...

So it's probably safe to increase this value. Anyway I don't think this value should be enforced by this option but rather by the memory available/allocated to the Node process. If you are working on large data (and you are allocating the right amount of memory to your Node process) then the default value of 1 megabytes is arbitrary and does not make much sense.

I agree, I'm not very familiar with node, just a few hacks but it seems that you can get the amount of memory currently allocated in a node process using process.memoryUsage() it seems to output both how much heap is available and how much is allocated.

{
 "rss": 4935680,
 "heapTotal": 1826816,
 "heapUsed": 650472,
 "external": 49879,
 "arrayBuffers": 9386
}

Thus, set the maxBuffer option more dynamically as you said but still make sure not to do FATAL ERROR ?

... And if it still works, there seems to be a flag to increase past the ~1.4GB limit using --max-old-space-size.

Maybe this is nonsens for you :)