ngageoint / hootenanny

Hootenanny conflates multiple maps into a single seamless map.
GNU General Public License v3.0
353 stars 74 forks source link

No error message specific to > 50k node geojson ingest #3133

Closed jackgrossman18 closed 5 years ago

jackgrossman18 commented 5 years ago

I'm in the process of allowing the import Multi-dataset to ingest multiple geojsons in the ui, ref hoot-ui ticket and I'm running into the issue from a few weeks ago that geojson's over 50,000 nodes are not accepted.

Would it be possible to return a more specific error message than the one that is currently being returned (error message example below)

{
jobId: "b4cec0cf-b884-4e7c-b999-f52d0d638757",
status: "failed",
statusDetail: "Command with ID = 284 caused the failure.",
percentcomplete: 0,
lasttext: "Command with ID = 284 caused the failure.",
commandDetail: [
{
command: ""hoot" "convert" "--info" "-D" "hootapi.db.writer.overwrite.map=true" "-D" "api.db.email=2635726@hootenanny" "/home/vagrant/hoot/userfiles/tmp/upload/b4cec0cf-b884-4e7c-b999-f52d0d638757/building-points.geojson" "hootapidb://${HOOTAPI_DB_USER}:${HOOTAPI_DB_PASSWORD}@${HOOTAPI_DB_HOST}:${HOOTAPI_DB_PORT}/${HOOTAPI_DB_NAME}/building-points.geo"",
exitCode: -1,
finish: 1555935156264,
id: 284,
jobId: "b4cec0cf-b884-4e7c-b999-f52d0d638757",
start: 1555935059661,
stderr: "Error running convert: Error parsing JSON: expected ',' or '}' (line 1) ",
stdout: "12:10:59.993 INFO ...hoot/core/io/DataConverter.cpp( 184) Converting /home/vagrant/hoot/userfiles/tmp/upload/b4cec0cf-b884-4e7c-b999-f52d0d638757/building-points.geojson to hootapidb://hoot:hoottest@localhost:5432/hoot/building-points.geo... 12:11:00.000 INFO ...ot/core/io/ElementStreamer.cpp( 80) Unable to stream I/O due to input: 7/building-points.geojson and/or output: /hoot/building-points.geo 12:11:00.000 INFO ...ore/io/OsmMapReaderFactory.cpp( 169) Loading map from 884-4e7c-b999-f52d0d638757/building-points.geojson... "
}
]
}

After failure, the following capabilities response is sent as well.

<osm attribution="?" copyright="?" generator="Hootenanny server" license="?" version="0.6">
<api>
<version maximum="0.6" minimum="0.6"/>
<area maximum="0.25"/>
<waynodes maximum="2000"/>
<changesets maximum_elements="50000"/>
<timeout seconds="3600"/>
<status api="online" database="online" gpx="offline"/>
</api>
</osm>

The message seems to suggest that when services are in the process of uploading two geojsons that together add up to more than 50k nodes, the second geojson is immediately truncated and then services interpret that file as malformed geojson, thus leading to the parsing error in the message I posted above.

If a > 50k nodes specific error message response is sent, it may be easier to add a warning message into the UI to inform users that the size of the files they're attempting to upload is too large

jackgrossman18 commented 5 years ago

@mattjdnv

matthias1231 commented 5 years ago

This issue was not related to node count but caused by the coordinate format (x,y,z instead of just x,y)