search

mmomtchev / node-gdal-async

Node.js bindings for GDAL (Geospatial Data Abstraction Library) with full async support
https://mmomtchev.github.io/node-gdal-async/
Apache License 2.0
129 stars 26 forks source link

Segmentation fault #90

Closed zdila closed 1 year ago

zdila commented 1 year ago

Hello,

I am getting Segmentation fault when using this library with https://github.com/FreemapSlovakia/dem-iron/tree/master and DTM Croatia 20m v1 by Sonny.tif from https://drive.google.com/drive/folders/1U152Bg0G1au_74ppTkZ3uNBdG-bMCXgN

I am running it on Debian 12 and Node 18 with gdal-async of the latest version on npm. Also tried older versions of Node and the library.

With Node 16 and gdal-async 3.1 I am getting

double free or corruption (out)
Aborted
mmomtchev commented 1 year ago

I confirm the crash, it is due to the fact that you are trying to read more than 4GB of data in a single operation. gdal-async uses Node.js NAN, the older Node.js API, which does not accept 64 bit integers. This means that anything over 4GB will have to use a floating point buffer size which is obviously not without risks. If I am able to come up with a safe implementation, I will extend the buffer limit to 2^53 - 1, JS max safe integer, or otherwise I will add a safety check at 4GB.

mmomtchev commented 1 year ago

TypedArrays in Node.js seem to be limited to 4G elements:

$ node -e "new Uint8Array(66000 * 66000)"
[eval]:1
new Uint8Array(66000 * 66000)
^

RangeError: Invalid typed array length: 4356000000
    at new Uint8Array (<anonymous>)
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:307:38)
    at node:internal/process/execution:79:19
    at [eval]-wrapper:6:22
    at evalScript (node:internal/process/execution:78:60)
    at node:internal/main/eval_string:28:3

Node.js v18.16.0

The underlying V8 ArrayBuffer has recently been extended to handle more than 4GB of data, but at the moment JS arrays are limited to 4G elements. So the hard limit for the amount of data retrieved in a single read or write operation will be 4G values for now - which depending on the element type might be more than 4GB on recent Node.js versions.