tulsawebdevs / django-multi-gtfs

Django app to import and export General Transit Feed Specification (GTFS)
http://tulsawebdevs.org/
Apache License 2.0
50 stars 32 forks source link

Migration error 2 #27

Closed araichev closed 10 years ago

araichev commented 10 years ago

Using multigtfs 0.3.3 on my local machine works great. Setting it up on my production web server on Webfaction gives me the migration error below. Any tips on what's going wrong?

$ python manage.py migrate multigtfs
Running migrations for multigtfs:
 - Migrating forwards to 0016_add_20140217_optional_fields.
 > multigtfs:0001_initial
 > multigtfs:0002_auto__chg_field_fare_transfers
 > multigtfs:0003_convert_fare_transfers
 > multigtfs:0004_auto_add_geopoint_fields
 > multigtfs:0005_migrate_lat_lon_to_point
 > multigtfs:0006_auto_del_lat_lon_fields
 > multigtfs:0007_rm_geography_points
Error in migration: multigtfs:0007_rm_geography_points
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 232, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 307, in migrate_many
    result = self.migrate(migration, database)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 132, in migrate
    result = self.run(migration, database)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 116, in run
    return self.run_migration(migration, database)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 86, in run_migration
    migration_function()
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/migration/migrators.py", line 62, in <lambda>
    return (lambda: direction(orm))
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/multigtfs/migrations/0007_rm_geography_points.py", line 26, in forwards
    db.alter_column('shape_point', 'point', field)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 47, in _cache_clear
    return func(self, table, *args, **opts)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 509, in alter_column
    "type": self._db_type_for_alter_column(field),
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/postgresql_psycopg2.py", line 93, in _db_type_for_alter_column
    return self._db_positive_type_for_alter_column(DatabaseOperations, field)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 1135, in _db_positive_type_for_alter_column
    super_result = super(klass, self)._db_type_for_alter_column(field)
  File "/home/araichev/.virtualenvs/corridor_explorer/lib/python2.7/site-packages/south/db/generic.py", line 435, in _db_type_for_alter_column
    return field.db_type()
TypeError: db_type_for_postgis() takes exactly 2 arguments (1 given)
jwhitlock commented 10 years ago

This is probably a mismatch between my installed South version (0.8.4) and yours. I can't reproduce with Django 1.5 / 1.6, or South 0.7.6 / 0.8.4.

Can you post the results of pip freeze?

jwhitlock commented 10 years ago

If a south upgrade is not possible, or is not the issue, this migration could be skipped and applied manually. Something like:

$ python manage.py dbshell
=> ALTER TABLE "shape_point" ALTER COLUMN "point" TYPE geometry(POINT,4326) USING ST_Transform(point::geometry, 4326), ALTER COLUMN "point" SET NOT NULL, ALTER COLUMN "point" DROP DEFAULT;
=> ALTER TABLE "stop" ALTER COLUMN "point" TYPE geometry(POINT,4326) USING ST_Transform(point::geometry, 4326), ALTER COLUMN "point" SET NOT NULL, ALTER COLUMN "point" DROP DEFAULT;
=> \q
$ python manage.py migrate multigtfs 0007 --fake
$ python manage.py migrate multigtfs
araichev commented 10 years ago

I just upgraded to South 0.8.4, tried to migrate again, and got the same error message (but with slightly different line numbers). Here's my pip freeze:

$ pip freeze
Django==1.6.2
MarkupSafe==0.18
Shapely==1.3.0
South==0.8.4
anyjson==0.3.3
django-crispy-forms==1.4.0
django-debug-toolbar==1.0.1
django-jsonfield==0.9.12
django-picklefield==0.3.1
django-registration==1.0
fastkml==0.5
ipython==1.0.0
jsonfield==0.9.20
jsonpickle==0.7.0
multigtfs==0.3.3
psycopg2==2.5.2
pygeoif==0.4.1
pyproj==1.9.3
python-dateutil==2.1
pytz==2013.7
six==1.3.0
sqlparse==0.1.11
utm==0.3.1
wsgiref==0.1.2

I also tried the dbshell commands above and got the following errors:

corridor_explorer_db=> ALTER TABLE "shape_point" ALTER COLUMN "point" TYPE geometry(POINT,4326) USING ST_Transform(point::geometry, 4326), ALTER COLUMN "point" SET NOT NULL, ALTER COLUMN "point" DROP DEFAULT;
ERROR:  type modifier is not allowed for type "geometry"
corridor_explorer_db=> ALTER TABLE "stop" ALTER COLUMN "point" TYPE geometry(POINT,4326) USING ST_Transform(point::geometry, 4326), ALTER COLUMN "point" SET NOT NULL, ALTER COLUMN "point" DROP DEFAULT;
ERROR:  type modifier is not allowed for type "geometry"

But when i faked migration 0007, i got no error messages and was able to upload a feed. Does that make sense? I don't know enough about South to tell.

jwhitlock commented 10 years ago

Migration 0007 is a weird (and probably invalid) one. It is supposed to convert geography points to geometry points in place. However, I think geography vs. geometry is handled on the Django side, so maybe it is failing because your points were always geometry points. I tried geography points in my own database, then decided not to use them.

Maybe this is a postgres / postgis version issue. I think the geometry type was added in postgis 2.0, and the type statement in postgres 8.3. Here's my versions:

$ psql -V
psql (PostgreSQL) 9.3.4
$ psql -d my_db_name
# SELECT PostGIS_version();
 2.1 USE_GEOS=1 USE_PROJ=1 USE_STATS=1

# SELECT PostGIS_full_version();
POSTGIS="2.1.1 r12113" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

The PostGIS_full_version may require superuser permissions - otherwise manage.py dbshell would work.

I didn't see anything that worried me in your pip freeze. Here's my pip freeze:

Django==1.6.2
South==0.7.6
Werkzeug==0.9.4
django-extensions==1.3.3
django-nose==1.2
flake8==2.1.0
ipdb==0.8
ipython==1.2.1
mccabe==0.2.1
-e git+git@github.com:tulsawebdevs/django-multi-gtfs.git@1f40bf44a38ef19b966251f4d12ec655a781a686#egg=multigtfs-origin/develop
nose==1.3.1
pep8==1.4.6
psycopg2==2.5.2
pyflakes==0.7.3
six==1.5.2
wsgiref==0.1.2

I think I'll change the migration to be OK if a TypeError is raised.

araichev commented 10 years ago

Aha, the development server on my local machine, on which everything works fine, has these versions of PostgreSQL and PostGIS:

$ psql -V
psql (PostgreSQL) 9.3.1
$ psql -d <database name>
psql (9.3.1)
Type "help" for help.

# SELECT PostGIS_full_version();
                                                                          postgis_full_version                                                                           
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 POSTGIS="2.1.0 r11822" GEOS="3.3.8-CAPI-1.7.8" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.0, released 2013/04/24" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER
(1 row)

whereas my production server on Webfaction, on which i get errors, has older versions:

$ psql -V
psql (PostgreSQL) 9.1.13
$ psql -d <database name>
# SELECT PostGIS_full_version();
                                         postgis_full_version                                          
-------------------------------------------------------------------------------------------------------
 POSTGIS="1.5.3" GEOS="3.3.8-CAPI-1.7.8" PROJ="Rel. 4.7.1, 23 September 2009" LIBXML="2.7.6" USE_STATS
(1 row)

So it probably is a PostGIS version issue.

jwhitlock commented 10 years ago

Knowing that it is PostGIS 1.5, I found this article:

http://geeohspatial.blogspot.com/2013/03/backwards-compatibility-of-geometry.html

That article gives several ways to set the column to a geometry in SRID 4326, and one simple but possibly risky way. Unfortunately, I don't know enough PostGIS to know if the migration is even necessary. It also looks like you're in for some pain upgrading to PostGIS 2.1, if you decide to do it someday.