Failed to import Portland's November feed

[araichev]

Hi folks,

Today i used multigtfs to import Portland's November feed (available at http://www.gtfs-data-exchange.com/agency/trimet/) and got the error below.

Is the problem that multigtfs doesn't handle feeds, such as Portland, that have a calendar_dates.txt file but no calendar.txt file? Such feeds are acceptable according to the GTFS, as described at https://developers.google.com/transit/gtfs/reference#calendar_dates_fields.

---

Traceback (most recent call last): File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) psycopg2.IntegrityError: null value in column "start_date" violates not-null constraint DETAIL: Failing row contains (117, 13, C.404, t, t, t, t, t, t, t, null, null, {}).

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/init.py", line 399, in execute_from_command_line utility.execute() File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/init.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv self.execute(_args, _options.dict) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/core/management/base.py", line 285, in execute output = self.handle(_args, _options) File "/Users/araichev/gtfse/explorer/management/commands/import_feed.py", line 45, in handle feed.import_gtfs(feed_file) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/feed.py", line 109, in import_gtfs count = klass.import_txt(table, self) or 0 File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/base.py", line 243, in import_txt fields[name_map[column_name]] = val_mapcolumn_name File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/multigtfs/models/base.py", line 149, in get_instance kwargs).id File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/manager.py", line 157, in create return self.get_queryset().create(kwargs) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/query.py", line 322, in create obj.save(force_insert=True, using=self.db) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 545, in save force_update=force_update, update_fields=update_fields) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 573, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 654, in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/base.py", line 687, in _do_insert using=using, raw=raw) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/manager.py", line 232, in _insert return insert_query(self.model, objs, fields, **kwargs) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/query.py", line 1514, in insert_query return query.get_compiler(using=using).execute_sql(return_id) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql cursor.execute(sql, params) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/utils.py", line 99, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/utils/six.py", line 549, in reraise raise value.with_traceback(tb) File "/Users/araichev/.virtualenvs/gtfse/lib/python3.4/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: null value in column "start_date" violates not-null constraint DETAIL: Failing row contains (117, 13, C.404, t, t, t, t, t, t, t, null, null, {}).

[jwhitlock]

I can reproduce, I'll take a closer look at the feed. My guess is that service 'C.404' doesn't appear in calendar.txt.

[jwhitlock]

OK, the Porland feed has no calendar.txt which is required in the GTFS spec, but does have calendar_dates.txt, which is optional. That's a strange choice, but this conversation might be relevant.

[jwhitlock]

And now I see what you meant. In the docs for calendar_dates.txt:

The calendar_dates table allows you to explicitly activate or disable service IDs by date. You can use it in two ways.

• Recommended: Use calendar_dates.txt in conjunction with calendar.txt, where calendar_dates.txt defines any exceptions to the default service categories defined in the calendar.txt file. If your service is generally regular, with a few changes on explicit dates (for example, to accomodate special event services, or a school schedule), this is a good approach.
• Alternate: Omit calendar.txt, and include ALL dates of service in calendar_dates.txt. If your schedule varies most days of the month, or you want to programmatically output service dates without specifying a normal weekly schedule, this approach may be preferable.

I'll have to special-case this usage, but it should be possible. We'll probably need a schema migration.

[araichev]

Yep, that's what i meant. Darn special cases. They're inevitable with the complexity of GTFS, i reckon.

Sorry for my late reply. I somehow messed up my Github notification settings and was notified of your comments.