Closed vicchi closed 4 years ago
@vicchi unfortunately, I don't use nor know anything about how Grunt works. I am wondering if Grunt is somehow hijacking the STDIO streams so STDERR doesn't output to the console. What is this.files
object look like in this context?
Grunt should be just plain Node.js. The fact that my standalone code works fine, makes me suspect that this isn't a Grunt problem per se, but a synchronous vs. asynchronous problem.
I could test this via Grunt's async()
support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr
or a way of working out how to deal with the data that ogr2ogr.exec
passes to its callback so I can get my final destination files written (and zipped if appropriate).
What I think is happening is that Grunt is assuming that ogr2ogr
runs synchronously and isn't waiting for the (internal) callbacks that ogr2ogr
uses to process the input and output files, so effectively Grunt finishes up whilst ogr2ogr
is still doing "stuff". If that makes sense.
Any suggestions for how to synchronise with ogr2ogr
actually completing?
Also, the contents of this.files
are
[
{
"src": [],
"dest": "data/dest/geojson/ne_110m_admin_0_sovereignty.geojson",
"orig": {
"src": [
"data/src/ne_110m_admin_0_sovereignty.shp"
],
"dest": "data/dest/geojson/ne_110m_admin_0_sovereignty.geojson"
}
}
]
... but I don't think this is a file specification problem.
I could test this via Grunt's async() support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr or a way of working out how to deal with the data that ogr2ogr.exec passes to its callback so I can get my final destination files written (and zipped if appropriate).
Streams returned from calling .stream()
from ogr2ogr will emit a close
event when all the processing is finished that you can listen for.
I could test this via Grunt's async() support, but to do that, I'd need either a way of getting a final, task completed, callback back from ogr2ogr or a way of working out how to deal with the data that ogr2ogr.exec passes to its callback so I can get my final destination files written (and zipped if appropriate).
Streams returned from calling .stream() from ogr2ogr will emit a close event when all the processing is finished that you can listen for.
Thanks. I'll give that a try. I'm offline travelling for a few weeks now but I'll give it a go when I'm back and online.
Progress. Had a spare few minutes before getting on a plane. Forcing Grunt to wait for the completion of ogr2ogr2
via the stream's close
event works perfectly. If you're interested, the key is using this.async()
and then calling the return value in the close
event handler. See below.
'use strict';
module.exports = function(grunt) {
var ogr2ogr = require('ogr2ogr');
var fs = require('fs');
var path = require('path');
// Please see the Grunt documentation for more information regarding task
// creation: http://gruntjs.com/creating-tasks
grunt.registerMultiTask('ogr2ogr', 'ogr2ogr wrapper task for Grunt', function() {
grunt.log.debug(JSON.stringify(this.files, null, 2));
// Merge task-specific and/or target-specific options with these defaults.
var options = this.options({
format: 'GeoJSON',
// skipFailures: true,
// projection: 'EPSG:4326',
// options: null
// timeout: 15000
});
var self = this;
// Iterate over all specified file groups.
this.files.forEach(function(f) {
var src = f.src.filter(function(filepath) {
if (!grunt.file.exists(filepath)) {
grunt.log.warn('Source file "' + filepath + '" not found.');
return false;
}
else {
return true;
}
}).map(function(filepath, i) {
grunt.file.mkdir(path.dirname(f.dest));
var done = self.async();
var stream = ogr2ogr(filepath).format(options.format).stream();
stream.on('close', function() {
done();
});
stream.pipe(fs.createWriteStream(f.dest));
});
});
});
};
I've loaded the output GeoJSON and Shapefile up in QGIS and all looks good. One final and unrelated question.
$ file ne_110m_admin_0_sovereignty.zip
ne_110m_admin_0_sovereignty.zip: Zip archive data, at least v2.0 to extract
$ unzip ne_110m_admin_0_sovereignty.zip
Archive: ne_110m_admin_0_sovereignty.zip
inflating: OGRGeoJSON.dbf
inflating: OGRGeoJSON.prj
inflating: OGRGeoJSON.shp
inflating: OGRGeoJSON.shx
Is there any way to specify the name of the (compressed) shapefile (as opposed to the resultant ZIP archive) to be something other than OGRGeoJSON
?
@vicchi for your last question you can change the name of the output file by using the -nln
(ie. new layer name) flag
for instance:
var shapefile = ogr2ogr('/path/to/spatial/file.geojson')
.format('ESRI Shapefile')
.options(['-nln', 'output_name'])
.skipfailures()
.stream()
shapefile.pipe(fs.createWriteStream('/shapefile.zip'))
Closing due to age, reopen if issue persists.
I'm trying to produce a Grunt plugin that wraps
ogr2ogr
functionality so I can use it without additional coding in my workflow. But I'm getting empty output files created, with no apparent errors or warnings.As a standalone (and shabby) piece of code, the following works fine ...
However, the same code when invoked from within Grunt's environment produces no errors or warnings and while an output file is produced, it's empty.
Adding some debug tracing shows that the source files exist and from the empty output I can see the right destination file is being created. But I'm now running into a brick wall of what to do next.
Thoughts and suggestions are welcomed!
Thanks