Handle case where bill rolls over to new legislative session

[hancush]

Currently, the importer considers legislative session when resolving bills for import, however bills can carry over to the next legislative session. When this happens, the bill no longer resolves because the related legislative session has changed, and pupa tries to insert it is a new bill. We impose a unique constraint on slug (populated from identifier) in Councilmatic, so the import subsequently fails.

Currently, bills can only be related to one legislative session, but in some jurisdictions, e.g., Metro, they can be associated with many.

How should pupa and Councilmatic behave in this instance? Can we assume that identifier is unique across all sessions, i.e., safely update pupa's resolution logic so legislative session is updated on an existing bill? Or should we account for the possibility that an identifier is only unique within a session? In that case, we'd need to update our slug logic and unique constraint to take identifier and session together.

Thoughts, @fgregg?

[hancush]

Related stacktrace:

UniqueViolation: duplicate key value violates unique constraint "councilmatic_core_bill_slug_ecb9ca6b_uniq"
DETAIL:  Key (slug)=(2021-0434) already exists.

  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
IntegrityError: duplicate key value violates unique constraint "councilmatic_core_bill_slug_ecb9ca6b_uniq"
DETAIL:  Key (slug)=(2021-0434) already exists.

  File "pupa/importers/base.py", line 291, in import_item
    obj = self.model_class.objects.create(**data)
  File "django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "django/db/models/base.py", line 744, in save
    force_update=force_update, update_fields=update_fields)
  File "django/db/models/base.py", line 793, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "django/dispatch/dispatcher.py", line 175, in send
    for receiver in self._live_receivers(sender)
  File "django/dispatch/dispatcher.py", line 175, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "councilmatic_core/signals/handlers.py", line 76, in create_councilmatic_bill
    cb.save_base(raw=True)
  File "django/db/models/base.py", line 782, in save_base
    force_update, using, update_fields,
  File "django/db/models/base.py", line 873, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "django/db/models/base.py", line 911, in _do_insert
    using=using, raw=raw)
  File "django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "raven/contrib/django/client.py", line 127, in execute
    return real_execute(self, sql, params)
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
DataImportError: duplicate key value violates unique constraint "councilmatic_core_bill_slug_ecb9ca6b_uniq"
DETAIL:  Key (slug)=(2021-0434) already exists.
 while importing {'identifier': '2021-0434', 'title': 'Restricted View', 'classification': ['bill'], 'subject': [], 'extras': {'restrict_view': True, 'plain_text': '', 'rtf_text': ''}, 'legislative_session_id': UUID('4eb5c351-7c68-4a28-9eb5-6fbbf27ce294'), 'from_organization_id': 'ocd-organization/46e47853-7856-4ae1-9901-8596e27f01bb'} as <class 'opencivicdata.legislative.models.bill.Bill'>
  File "bin/pupa", line 8, in <module>
    sys.exit(main())
  File "pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "pupa/cli/commands/update.py", line 329, in do_handle
    report['import'] = self.do_import(juris, args)
  File "pupa/cli/commands/update.py", line 219, in do_import
    report.update(bill_importer.import_directory(datadir))
  File "pupa/importers/base.py", line 197, in import_directory
    return self.import_data(json_stream())
  File "pupa/importers/base.py", line 234, in import_data
    obj_id, what = self.import_item(data)
  File "pupa/importers/base.py", line 294, in import_item
    self.model_class))

[fgregg]

i think the concept of legislative session includes the idea that bills cannot be carried over. so, i would say that these are not legislative sessions. are we using "legislative session" anywhere in the metro site to do anything for us?

[hancush]

You're right – for Metro, the legislative session is actually a fiscal year, IIRC. We just it out of the search index, so no, I don't see that we are using it anywhere meaningful, @fgregg. What are you thinking?

[fgregg]

pretty sure that OCD requires a legislative session, but we can just have one legislative session.