shz file saved as a directory on linux, breaking stream processing

[marknutrien]

I've been working with this package to convert GeoJson to Shapefile zip stream and it works great....just until I try to package it up in a Node.js Debian docker container. The error I'm getting when trying to convert a stream into a Buffer. It reports an EISDIR which is the stream trying to process it as a file while the OS reports back that it is a directory. I've verified that the .shz file being saved in the linux container as a directory, whereas in my MACOS environment it is saved as a file. The primary difference I can see is that the MAC installed a 3.2 version of ogr2ogr (when I run ogrinfo --version) whereas the linux version installed version 2.1. Not sure if that is the culprit but it seems like some accommodation may need to be made with this package to work in Docker. Here is the code that works on MacOS and not on Linux:

stream2Buffer(inStream) {
    return new Promise((resolve, reject) => {
      const buf = [];

      inStream.on('data', (chunk) => buf.push(chunk));
      inStream.on('end', () => resolve(Buffer.concat(buf)));
      inStream.on('error', (err) => reject(err));
    });
  }
 async convertGeoJsonToShapeFile({ geometry }) {
    const ogr2ogr = Ogr2ogr.default;

    const { stream } = await ogr2ogr(geometry, {
      format: 'ESRI Shapefile'
    });
    const buffer = await this.stream2Buffer(stream);
    return buffer;
 }

Or, if I'm missing something simple to make it work, happy to adjust.

Thanks for making this excellent library!

[wavded]

I think its the GDAL version you have installed on Linux. SHZ support is a newer file type that GDAL added support for. Are you able to get a 3.x version of GDAL installed on Linux?

[marknutrien]

Yeah, that's kinda what I was afraid of. After spending the better part of the day trying to figure out how to get the 3.x version working, it seems that support for GDAL 3.x on Debian is not supported except for in their "unstable" Debian packages (see https://gdal.org/download.html#binaries) which would never fly at my client. They use the nodeJS Debian docker images as its build source so that isn't negotiable. I did try to see if I could use Anaconda or other ways to get GDAL injected into the docker image but it was getting pretty messy. I ultimately hacked around it by modifying the convertGeoJsonToShapeFile with:

async convertGeoJsonToShapeFile({ geometry }) {
    const ogr2ogr = Ogr2ogr.default;
    const { stream } = await ogr2ogr(geometry, {
      format: 'ESRI Shapefile'
    });

    let buffer;
    try {
      // the normal way to process on a dev machine
      buffer = await this.stream2Buffer(stream);
    } catch (e) {
      // docker ogr2ogr which is v2.1 doesn't save it as a file
      // but rather  as a directory, so zip the directory
      const archive = archiver('zip');
      archive.directory(stream.path, false);
      archive.finalize();
      buffer = await this.stream2Buffer(archive);
    }

    return buffer;
  }

Perhaps this repo could support the 2.x output by evaluating the resulting stream fromres.stream = createReadStream(this.outputPath) and if the length of the stream was zero, send it through the createZipStream() function instead?

[wavded]

This current version 3.x really is designed around GDAL 3 support in this case. I intentionally made the API as close to the ogr2ogr tool as possible in 3.x to avoid various issues we had in the past (so this lib's support is tied to ogr2ogr's support rather than trying to make special cases). The 2.x version should support your use case for now until you are able to upgrade.

You probably have already went through this in your mind but GDAL does have official docker containers that have the latest version and they document using the unstable repo for their release on Debian platforms. Of course you can always build from source too.

https://gdal.org/download.html

[marknutrien]

Yeah, no prob. that makes sense. 99.9% that their corporate security wouldn't allow the unstable docker builds and there is a lot of other intermediate build steps that are required with their CI/CD stuff. I'll keep the code I have for now - like most projects, on to the next thing. Thanks! I'll close this issue.