search

ngageoint / hootenanny

Hootenanny conflates multiple maps into a single seamless map.
GNU General Public License v3.0
353 stars 74 forks source link

FGDB export failing for MGCP, TDS61 on merged Latvia and Brazil POI data. #258

Closed mikejeffe closed 8 years ago

mikejeffe commented 8 years ago

Unable to export merged Brazil Sao Paulo osm/geonames and merged Latvia osm/geonames data as .gdb using either mgcp or tds61 schemas. So far these are the only two files with the export issue but it could occur with similar osm/geonames merged data. This is occurring on the rc and develop branch. Let me know if you need the source data. Data are ingested on release branch on selma. Catalina.out log can be viewed on the server for debugging.

mikejeffe commented 8 years ago

Looks like both the Latvia and Brazil osm datasets have the source:datetime tag in this format: 2015-03-14T13:58:30.000Z

mattjdnv commented 8 years ago

@mjeffe98 The code was fixed in the develop branch a while ago.

Using LatviaCropped_OSM_REF1.osm from gis-data, it has source:datetime tags like this:

<tag k='source:datetime' v='2015-03-30T15:31:53.000Z' />
<tag k='source:datetime' v='2012-02-12T02:28:00.000Z' />
<tag k='source:datetime' v='2014-09-20T17:26:09.000Z' />
<tag k='source:datetime' v='2015-07-28T23:53:44.000Z' />
<tag k='source:datetime' v='2015-02-21T23:30:30.000Z' />

Exporting to TDSv61 with debugging turned on gives this type of output. Note that the source:datetime tag has been converted to what TDS expects (ZI001_SDV).

In Geometry: Point  In Element Type: Node
In Tags: REF1: :001c85:
In Tags: error:circular: :15:
In Tags: hoot:status: :invalid:
In Tags: amenity: :cafe:
In Tags: source:datetime: :2012-07-14T14:11:37.000Z:
TableName 0: StructurePnt  FCode: AL013  Geom: Point
Out Attrs:F_CODE: :AL013:
Out Attrs:FFN: :572:
Out Attrs:ZI001_SDV: :2012-07-14T14:11:37Z:
Out Attrs:UFI: :a9813915-ee29-488d-ab6f-4ca0d94ab2dc:
Out Attrs:FCSUBTYPE: :100083:
Out Attrs:TEL: :undefined:
Out Attrs:HEI: :undefined:
mattjdnv commented 8 years ago

Talking with @mjeffe98 , we have worked out that for this file, the problem is with other text fields during export. From the TDS spec, some of the text fields have set character lengths (E.g. ZI005_FNA is 200 characters). I'll modify the translation code to validate text fields with set lengths prior to export.

mattjdnv commented 8 years ago

This is fixed by #271 so I'm closing this ticket.