tulsawebdevs / django-multi-gtfs

Django app to import and export General Transit Feed Specification (GTFS)
http://tulsawebdevs.org/
Apache License 2.0
51 stars 33 forks source link

Can't export a non-ascii feed in Python 2 #34

Closed jwhitlock closed 10 years ago

jwhitlock commented 10 years ago

When exporting a stop with a bad description (utf-8 "The Delta Caf‚\xc2\x82"), export fails:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/john/.virtualenvs/tulsa-transit-google/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/john/src/django-multi-gtfs/multigtfs/management/commands/exportgtfs.py", line 80, in handle
    feed.export_gtfs(out_name)
  File "/Users/john/src/django-multi-gtfs/multigtfs/models/feed.py", line 158, in export_gtfs
    klass._filename, content.count('\n') - 1,
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 72255: ordinal not in range(128)

Under Python 2, content is a UTF-8 encoded byte string, and '\n' is a unicode string. content.count('\n') attempts to decode it as an ascii string first. Under Python 3, both content and '\n' are unicode strings.