ngageoint / hootenanny

Hootenanny conflates multiple maps into a single seamless map.
GNU General Public License v3.0
356 stars 74 forks source link

Handle fields with int64 values #1487

Closed mattjdnv closed 7 years ago

mattjdnv commented 7 years ago

Some values are larger then the long int that we are using to store them.

E.g. The long int being used must be 32-bit and has a max of 2,147,483,647 the osm spec for id used 64-bit int From the RenderDb schema: { name:'osm_id',desc:'osm_id',type:'Integer',defValue:'-999999' },

Modify Hoot to handle "long long" integer fields.

brianhatchl commented 7 years ago

This was the initial error:

vagrant@vagrant-ubuntu-trusty-64:~/hoot$ hoot osm2ogr translations/RenderDb.js export_93dd78f0-161c-11e7-9870-39664411cc6e.osm test.gdb 14:23:56.553 INFO ...ore/io/OsmMapReaderFactory.cpp( 135) Loading map from export_93dd78f0-161c-11e7-9870-39664411cc6e.osm... 14:24:03.631 INFO ...hoot/core/schema/OsmSchema.cpp(1838) Loading translation files... Error running osm2ogr: Cannot convert value to long integer: 3623458488

Changing the schema type to "Real" got past this and to another error:

vagrant@vagrant-ubuntu-trusty-64:~/hoot$ hoot osm2ogr translations/RenderDb.js export_93dd78f0-161c-11e7-9870-39664411cc6e.osm test.gdb 14:39:39.425 INFO ...ore/io/OsmMapReaderFactory.cpp( 135) Loading map from export_93dd78f0-161c-11e7-9870-39664411cc6e.osm... 14:39:46.087 INFO ...hoot/core/schema/OsmSchema.cpp(1838) Loading translation files... 14:39:46.218 INFO ...cpp/hoot/core/io/OgrWriter.cpp( 302) Layer: Point not found. Creating layer... Error running osm2ogr: Layer creation failed. Point

A gdb output dir is created, but it seems empty:

vagrant@vagrant-ubuntu-trusty-64:~/hoot$ ogrinfo test.gdb INFO: Open of test.gdb' using driverFileGDB' successful.

mattjdnv commented 7 years ago

@brianhatchl Did you delete the FGDB after the first run? Hoot complains if the FGDB exists when it is trying to write to it - unless you set the "ogr.append.data" flag.

Changing the osm_id type from Integer to Real allowed the command to without dieing but it has these warnings in the shapefile export:

Out Tags :osm_id: :3836631579:
Out Tags :natural: :tree:
Out Tags :tags: :"natural"=>"tree","source:ingest:datetime"=>"2017-03-1T18:10:24.480Z",
"hoot:layername"=>"planet_osm_point","hoot:status"=>"Invalid":
12:44:09.887 WARN  ...?(  -1) CPL Error #: 1 Value 3836631579 of field osm_id of feature 1 not successfully written. Possibly due to too larger number with respect to field width

Looking in QGIS, the osm_id field has this value: "3836631579.000000000000000"

The weird thing is, writing to FGDB didn't have any warnings and has numbers like:
osm_id (Real) = 998952018

brianhatchl commented 7 years ago

I did not remove any output and used the same filename, so that was probably the cause of my error.

Is the data type of the id column only important for sorting purposes? Maybe we could just make it text.

mschicker commented 7 years ago

Merged #1490