./manage.py cities fails on postgis

[jpic]

Postgresql: 9.1.1 Postgis: 1.5 Geos: 3.3.0-1 Proj: 4.7.0 Django: 1.3,1 Distro: Arch

Settings:

CITIES_FILES = {
    'city': {
       'filename': 'cities5000.zip',
       'urls':     ['http://download.geonames.org/export/dump/'+'{filename}']
    },
}

CITIES_POSTAL_CODES = ['US','CA', 'FR']
CITIES_LOCALES = ['en', 'fr']
CITIES_PLUGINS = [
    'cities.plugin.postal_code_ca.Plugin',
]

Error when running "cities" command:

Traceback (most recent call last):
  File "./manage.py", line 20, in <module>
    execute_from_command_line()
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/srv/art_dev/art_dev_env/src/cities/cities/management/commands/cities.py", line 68, in handle
    func()
  File "/srv/art_dev/art_dev_env/src/cities/cities/management/commands/cities.py", line 494, in import_postal_code
    pc.save()
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/backends/util.py", line 34, in execute
    return self.cursor.execute(sql, params)
  File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0x00

Also here are some errors from shell_plus command:

Failed to import 'CountryAltNameFr' from 'cities' reason: 'module' object has no attribute 'CountryAltNameFr' Failed to import 'CountryAltNameEn' from 'cities' reason: 'module' object has no attribute 'CountryAltNameEn' Failed to import 'RegionAltNameFr' from 'cities' reason: 'module' object has no attribute 'RegionAltNameFr' Failed to import 'RegionAltNameEn' from 'cities' reason: 'module' object has no attribute 'RegionAltNameEn' Failed to import 'CityAltNameFr' from 'cities' reason: 'module' object has no attribute 'CityAltNameFr' Failed to import 'CityAltNameEn' from 'cities' reason: 'module' object has no attribute 'CityAltNameEn' Failed to import 'DistrictAltNameFr' from 'cities' reason: 'module' object has no attribute 'DistrictAltNameFr' Failed to import 'DistrictAltNameEn' from 'cities' reason: 'module' object has no attribute 'DistrictAltNameEn' Failed to import 'PostalCodeCA' from 'cities' reason: 'module' object has no attribute 'PostalCodeCA' Failed to import 'PostalCodeFR' from 'cities' reason: 'module' object has no attribute 'PostalCodeFR' Failed to import 'PostalCodeUS' from 'cities' reason: 'module' object has no attribute 'PostalCodeUS'

[Qarterd]

I'm not able to reproduce this. The traceback is showing that some postal code (probably FR) is failing to save to the db, but all FR codes import without issue for me. Try changing LOGGING.loggers.cities.level from 'INFO' to 'DEBUG' and run again (with --force --import=postal_code), it will show the last successfully imported line.

[jpic]

Thanks a lot for your support ! Sorry for the delay

Here is the log:

Added region 1: ZW.00.893482, Chiredzi District, Zimbabwe Region 1: Chipinge District: Cannot find region 0: ZW.00 Added region 1: ZW.00.893548, Chipinge District, Zimbabwe Region 1: Chimanimani District: Cannot find region 0: ZW.00 Added region 1: ZW.00.893811, Chimanimani District, Zimbabwe Region 1: Chegutu District: Cannot find region 0: ZW.00 Added region 1: ZW.00.894238, Chegutu District, Zimbabwe Region 1: Charter District: Cannot find region 0: ZW.00 Added region 1: ZW.00.894293, Charter District, Zimbabwe Region 1: Centenary District: Cannot find region 0: ZW.00 Added region 1: ZW.00.894459, Centenary District, Zimbabwe Added region 1: ZW.09.894700, Bulawayo District, Bulawayo, Zimbabwe Added region 1: ZW.07.894702, Bulilimamangwe District, Matabeleland South, Zimbabwe Region 1: Buhera District: Cannot find region 0: ZW.00 Added region 1: ZW.00.894711, Buhera District, Zimbabwe Region 1: Bubi District: Cannot find region 0: ZW.00 Added region 1: ZW.00.894741, Bubi District, Zimbabwe Region 1: Binga District: Cannot find region 0: ZW.00 Added region 1: ZW.00.895055, Binga District, Zimbabwe Region 1: Bindura District: Cannot find region 0: ZW.00 Added region 1: ZW.00.895060, Bindura District, Zimbabwe Region 1: Bikita District: Cannot find region 0: ZW.00 Added region 1: ZW.00.895080, Bikita District, Zimbabwe Region 1: Beitbridge District: Cannot find region 0: ZW.00 Added region 1: ZW.00.895268, Beitbridge District, Zimbabwe Downloading: cities5000.zip Building region index Importing city data Using settings for master master.art.yourlabs.org Using settings for master master.art.yourlabs.org Traceback (most recent call last): File "./clients/master/manage.py", line 23, in execute_from_command_line() File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/init.py", line 429, in execute_from_command_line utility.execute() File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/init.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv self.execute(_args, _options.dict) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute output = self.handle(_args, _options) File "/srv/art_dev/art_dev_env/src/cities/cities/management/commands/cities.py", line 68, in handle func() File "/srv/art_dev/art_dev_env/src/cities/cities/management/commands/cities.py", line 315, in import_city city.save() File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save self.save_base(using=using, force_insert=force_insert, force_update=force_update) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/base.py", line 553, in save_base result = manager._insert(values, return_id=update_pk, using=using) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/manager.py", line 195, in _insert return insert_query(self.model, values, **kwargs) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/query.py", line 1436, in insert_query return query.get_compiler(using=using).execute_sql(return_id) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql cursor = super(SQLInsertCompiler, self).execute_sql(None) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql cursor.execute(sql, params) File "/srv/art_dev/art_dev_env/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute return self.cursor.execute(query, args) django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0x00

[Qarterd]

I googled it, it appears that postgres 9.1 changed the default setting 'standard_conforming_strings' from off to on, and old versions of geodjango don't support on. You can either change that setting in your postgres config to off, or upgrade your django to the latest dev trunk (which I'm using, the trunk seems quite stable).

Alternatively, you could apply this patch to your geodjango: https://code.djangoproject.com/attachment/ticket/16778/postgis-adapter-2.patch

[jpic]

Thanks O PLENTY ! you are django-cities hero :)

I'll wait for Django 1.4 which is in 3 days, will reopen if that doesn't work.

[coderholic]

Confirm this is fixed in Django 1.4