Fails with non-ascii chars

[naoliv]

While trying to use ogr2osm witht this shapefile http://people.debian.org/~naoliv/misc/shape-ogr2osm.tar.bz2 I got this:

$ ./ogr2osm.py -p "+proj=utm +zone=23 +south +ellps=GRS67 +towgs84=-66.87,4.37,-38.52" ../SP_USINAS.SHP
running with ElementTree on Python 2.5+
Preparing to convert file '/tmp/d/SP_USINAS.SHP' to '/tmp/d/ogr2osm/SP_USINAS.osm'.
Will use the PROJ.4 string: +proj=utm +zone=23 +south +ellps=GRS67 +towgs84=-66.87,4.37,-38.52
Using default translations
Using default filterLayer
Using default filterFeature
Using default filterTags
Using default filterFeaturePost
Using default preOutputTransform
Parsing data
Merging points
Making list
Checking list
Outputting XML
Traceback (most recent call last):
  File "./ogr2osm.py", line 636, in <module>
    output()
  File "./ogr2osm.py", line 591, in output
1       ./SP_USINAS.DBF 
2       ./SP_USINAS.PRJ 
3       ./SP_USINAS.SBN 
4       ./SP_USINAS.SBX 
5       ./SP_USINAS.SHP
    f.write(etree.tostring(xmlobject))
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1127, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 821, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 940, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 933, in _serialize_xml
    v = _escape_attrib(v, encoding)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1091, in _escape_attrib
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 22: ordinal not in range(128)

Then installed python-lxml but it still fails:

$ ./ogr2osm.py -p "+proj=utm +zone=23 +south +ellps=GRS67 +towgs84=-66.87,4.37,-38.52" ../SP_USINAS.SHP
running with lxml.etree
Preparing to convert file '/tmp/d/SP_USINAS.SHP' to '/tmp/d/ogr2osm/SP_USINAS.osm'.
Will use the PROJ.4 string: +proj=utm +zone=23 +south +ellps=GRS67 +towgs84=-66.87,4.37,-38.52
Using default translations
Using default filterLayer
Using default filterFeature
Using default filterTags
Using default filterFeaturePost
Using default preOutputTransform
Parsing data
Merging points
Making list
Checking list
Outputting XML
Traceback (most recent call last):
  File "./ogr2osm.py", line 636, in <module>
    output()
  File "./ogr2osm.py", line 588, in output
    tag = etree.Element('tag', {'k':key, 'v':value})
  File "lxml.etree.pyx", line 2570, in lxml.etree.Element (src/lxml/lxml.etree.c:52973)
  File "apihelpers.pxi", line 156, in lxml.etree._makeElement (src/lxml/lxml.etree.c:11834)
  File "apihelpers.pxi", line 144, in lxml.etree._makeElement (src/lxml/lxml.etree.c:11709)
  File "apihelpers.pxi", line 299, in lxml.etree._initNodeAttributes (src/lxml/lxml.etree.c:13280)
  File "apihelpers.pxi", line 1369, in lxml.etree._utf8 (src/lxml/lxml.etree.c:22147)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

[simon04]

The following patch fixes the issue for the above shapefile as well as for my personal project:

a/ogr2osm.py b/ogr2osm.py
index 41541f3..d3d081d 100755
--- a/ogr2osm.py
+++ b/ogr2osm.py
@@ -387,7 +387,7 @@ def getLayerFields(layer):
 def getFeatureTags(ogrfeature, fieldNames):
     tags = {}
     for i in range(len(fieldNames)):
-        tags[fieldNames[i]] = ogrfeature.GetFieldAsString(i)
+        tags[fieldNames[i].decode('utf-8')] = ogrfeature.GetFieldAsString(i).decode('utf-8')
     return translations.filterTags(tags)

 def parseLayer(layer):

[simon04]

Your commit 8eb57cfcae743c3b348161f685e483e238288e1d has the disadvantage that writing translation files is more tricky. Consider the example

# -*- coding: utf-8 -*-
def translateAttributes(attrs):
    return {'name': u'ß' + attrs['NAME']}

• In 8eb57cfcae743c3b348161f685e483e238288e1d, this gives rise to UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)
• … whereas the example works in my solution

[pnorman]

What line is it raising that error on?

[simon04]

Traceback (most recent call last):
  File "/home/simon/src/ogr2osm/ogr2osm.py", line 639, in <module>
    parseData(data)
  File "/home/simon/src/ogr2osm/ogr2osm.py", line 345, in parseData
    parseLayer(translations.filterLayer(layer))
  File "/home/simon/src/ogr2osm/ogr2osm.py", line 402, in parseLayer
    parseFeature(translations.filterFeature(ogrfeature, fieldNames, reproject), fieldNames, reproject)
  File "/home/simon/src/ogr2osm/ogr2osm.py", line 419, in parseFeature
    feature.tags = getFeatureTags(ogrfeature, fieldNames)
  File "/home/simon/src/ogr2osm/ogr2osm.py", line 392, in getFeatureTags
    return translations.filterTags(tags)
  … Translation file, line where the concatenation takes place
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)

[pnorman]

I'm going to have to check that change against shift_jis shapefiles and existing translations (e.g. https://github.com/nyampire/ogr2osm/blob/master/translations/N03-admin_border.py)

[naoliv]

With this shapefile http://people.debian.org/~naoliv/misc/monster-shape.tar.bz2 and latest code I see another related problem:

$ ./ogr2osm.py -v ~/m/35DSE250GC_SIR.shp 
running with lxml.etree
Preparing to convert file '/home/naoliv/m/35DSE250GC_SIR.shp' to '/tmp/d/ogr2osm/35DSE250GC_SIR.osm'.
Will try to detect projection from source metadata, or fall back to EPSG:4326
Using default translations
Using default filterLayer
Using default filterFeature
Using default filterTags
Using default filterFeaturePost
Using default preOutputTransform
Parsing data
Detected projection metadata:
GEOGCS["SIRGAS 2000",
    DATUM["SIRGAS2000",
        SPHEROID["Geodetic_Reference_System_of_1980",6378137,298.2572221009113]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
Merging points
Making list
Checking list
Outputting XML
Traceback (most recent call last):
  File "./ogr2osm.py", line 642, in <module>
    output()
  File "./ogr2osm.py", line 610, in output
    tag = etree.Element('tag', {'k':unicode(key,'utf-8'), 'v':unicode(value, 'utf-8')})
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcd in position 4: unexpected end of data

[simon04]

Note that also QGIS has encoding problems, i.e., the attribute table shows some strange symbols.

On the GDAL side, general Unicode support isn't available: http://www.mail-archive.com/gdal-dev@lists.osgeo.org/msg01993.html

To come around this issue, the desired encoding could be given as commandline argument (default utf-8) …

[pnorman]

naoliv, for the second shapefile where there are still errors it should work if you use --encoding ISO8859-1