search

yocontra / node-gdal-next

Node.js bindings for GDAL (Geospatial Data Abstraction Library) [Fork]
https://contra.io/node-gdal-next/
Apache License 2.0
75 stars 35 forks source link

Implement a JS gate to the vsimem interface #39

Closed mmomtchev closed 3 years ago

mmomtchev commented 3 years ago

As an alternative to #4 , I propose that a gate to the vsimem interface is implemented, allowing to use gdal.open on a JS object of Buffer type Instead of relying on the rudimentary support in GDAL to make the network requests, a Node.js program which could build upon the rich JS ecosystem, will be much better suited to do the networking and leave the processing to GDAL

This is a very interesting JS-specific project and will be a useful addition to node-gdal-next

Alas, GDAL requires random-access I/O, so a Read/WriteStream can't be compatible - when using curl it will repeatedly ask the HTTP server for various parts of the file that is being read, often in a non-sequential manner In particular, GeoJSON is infamous for being impossible to incrementally parse (which is the reason behind GeoJSONSeq)

Still, when coupled with the native http module, this approach can have excellent performance - only the setup will be done by JS/V8 with the C++ native code taking care of the actual data transfers

mmomtchev commented 3 years ago

@andrewharvey @Ysgorg I have an initial version at https://github.com/mmomtchev/node-gdal-async/tree/vsimem You can call gdal.open on a Buffer object - everything works as for a normal file Persistence is handled, but not enforced - if the GC deletes your buffer, the application won't crash, but subsequent operations will fail with an I/O error I wonder if I shall keep a strong reference on the Buffer

If it works, it will probably be merged very soon

mmomtchev commented 3 years ago

This is ready for merging I added a reference to the Buffer object in the dataset in gdal.Dataset.buffer - this way you can be sure that the lifetime of the dataset won't exceed the lifetime of the buffer

yocontra commented 3 years ago

Shipped in 2.6.0