Adapt the app for Councilmatic 2.5

jeancochrane commented 5 years ago

Overview

Make sure the app is compatible with django-councilmatic v2.5. The major changes in this update include:

Support Django 2.0+
Refactor queries in send_notifications management command to use the ORM and the new OCD model structure
Write some basic tests for the app

Notes

This PR supercedes #33. The diff between the two branches was large, and conversation on that PR had gotten quite long, so I decided to open a fresh one.

Testing instructions

Make a virtualenv for the project and install it for testing with pip install -e .[tests]
Run pytest and confirm all tests pass

hancush commented 5 years ago

hm, @jeancochrane, i'm getting test failures, even after installing django-councilmatic as instructed:

(django-councilmatic-notifications) call-me-hank:django-councilmatic-notifications hannah$ pytest
================================================ test session starts ================================================
platform darwin -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
Django settings: tests.test_config (from ini file)
rootdir: /Users/hannah/projects/django-councilmatic-notifications, inifile: setup.cfg
plugins: django-3.5.1, mock-1.10.4
collected 13 items

tests/test_management_commands.py .F.....F...                                                                 [ 84%]
tests/test_views.py ..                                                                                        [100%]

===================================================== FAILURES ======================================================
__________________________________ test_find_bill_action_updates_skips_null_dates ___________________________________

self = <django.db.backends.utils.CursorWrapper object at 0x10f9b6320>
sql = 'SELECT "opencivicdata_billaction"."id", "opencivicdata_billaction"."bill_id", "opencivicdata_billaction"."organizatio...D ("opencivicdata_billaction"."date")::timestamp with time zone >= %s) ORDER BY "opencivicdata_billaction"."order" ASC'
params = ('ocd-bill/19cf632a-8323-4c8f-afb3-9d5294b2e7cd', datetime.datetime(2019, 7, 23, 15, 7, 45, 542694, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>))
ignored_wrapper_args = (False, {'connection': <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x10bf66e10>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x10f9b6320>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                return self.cursor.execute(sql)
            else:
>               return self.cursor.execute(sql, params)
E               psycopg2.errors.InvalidDatetimeFormat: invalid input syntax for type timestamp with time zone: ""

../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: InvalidDatetimeFormat

The above exception was the direct cause of the following exception:

new_bill = <Bill: test bill>
new_bill_actions = (<BillAction: test bill action on >, <BillAction: test bill action on >)

    @pytest.mark.django_db
    def test_find_bill_action_updates_skips_null_dates(new_bill, new_bill_actions):
        for bill_action in new_bill_actions:
            # Null dates are saved as empty strings in the OCD data model
            bill_action.date = ''
            bill_action.save()
        command = Command()
>       bill_action_updates = command.find_bill_action_updates([new_bill.id], minutes=15)

tests/test_management_commands.py:33:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
notifications/management/commands/send_notifications.py:209: in find_bill_action_updates
    for action in new_actions:
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:268: in __iter__
    self._fetch_all()
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:1186: in _fetch_all
    self._result_cache = list(self._iterable_class(self))
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:54: in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1065: in execute_sql
    cursor.execute(sql, params)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:68: in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
    return executor(sql, params, many, context)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: in _execute
    return self.cursor.execute(sql, params)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/utils.py:89: in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <django.db.backends.utils.CursorWrapper object at 0x10f9b6320>
sql = 'SELECT "opencivicdata_billaction"."id", "opencivicdata_billaction"."bill_id", "opencivicdata_billaction"."organizatio...D ("opencivicdata_billaction"."date")::timestamp with time zone >= %s) ORDER BY "opencivicdata_billaction"."order" ASC'
params = ('ocd-bill/19cf632a-8323-4c8f-afb3-9d5294b2e7cd', datetime.datetime(2019, 7, 23, 15, 7, 45, 542694, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>))
ignored_wrapper_args = (False, {'connection': <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x10bf66e10>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x10f9b6320>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                return self.cursor.execute(sql)
            else:
>               return self.cursor.execute(sql, params)
E               django.db.utils.DataError: invalid input syntax for type timestamp with time zone: ""

../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: DataError
_______________________________________ test_find_new_events_skips_null_dates _______________________________________

self = <django.db.backends.utils.CursorWrapper object at 0x10f839d68>
sql = 'SELECT "opencivicdata_event"."created_at", "opencivicdata_event"."updated_at", "opencivicdata_event"."extras", "openc...created_at" >= %s AND ("opencivicdata_event"."start_date")::timestamp with time zone >= %s) ORDER BY "start_time" DESC'
params = (datetime.datetime(2019, 7, 23, 15, 7, 45, 892043, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>), datetime.datetime(2019, 7, 23, 15, 22, 45, 892065, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>))
ignored_wrapper_args = (False, {'connection': <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x10bf66e10>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x10f839d68>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                return self.cursor.execute(sql)
            else:
>               return self.cursor.execute(sql, params)
E               psycopg2.errors.InvalidDatetimeFormat: invalid input syntax for type timestamp with time zone: ""

../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: InvalidDatetimeFormat

The above exception was the direct cause of the following exception:

new_events = (<Event: test event 1 ()>, <Event: test event 2 ()>)

    @pytest.mark.django_db
    def test_find_new_events_skips_null_dates(new_events):
        for event in new_events:
            # Null dates are saved as empty strings in the OCD data model
            event.start_date = ''
            event.save()
        command = Command()
>       found_events = command.find_new_events()

tests/test_management_commands.py:121:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
notifications/management/commands/send_notifications.py:385: in find_new_events
    for event in new_events_q:
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:268: in __iter__
    self._fetch_all()
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:1186: in _fetch_all
    self._result_cache = list(self._iterable_class(self))
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/query.py:54: in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/models/sql/compiler.py:1065: in execute_sql
    cursor.execute(sql, params)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:68: in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
    return executor(sql, params, many, context)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: in _execute
    return self.cursor.execute(sql, params)
../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/utils.py:89: in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <django.db.backends.utils.CursorWrapper object at 0x10f839d68>
sql = 'SELECT "opencivicdata_event"."created_at", "opencivicdata_event"."updated_at", "opencivicdata_event"."extras", "openc...created_at" >= %s AND ("opencivicdata_event"."start_date")::timestamp with time zone >= %s) ORDER BY "start_time" DESC'
params = (datetime.datetime(2019, 7, 23, 15, 7, 45, 892043, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>), datetime.datetime(2019, 7, 23, 15, 22, 45, 892065, tzinfo=<DstTzInfo 'America/Chicago' CDT-1 day, 19:00:00 DST>))
ignored_wrapper_args = (False, {'connection': <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x10bf66e10>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x10f839d68>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
                return self.cursor.execute(sql)
            else:
>               return self.cursor.execute(sql, params)
E               django.db.utils.DataError: invalid input syntax for type timestamp with time zone: ""

../../.virtualenvs/django-councilmatic-notifications/lib/python3.7/site-packages/django/db/backends/utils.py:85: DataError
======================================= 2 failed, 11 passed in 20.69 seconds ========================================

jeancochrane commented 5 years ago

@hancush Did you switch to the new branch (feature/jfc/councilmatic-2.5)? I removed the test test_find_bill_action_updates_skips_null_dates in this branch but it seems to be showing up in your traceback.

hancush commented 5 years ago

hhhhhmmmmmmmmmmmmm @ myself. thanks, @jeancochrane, you're right. sorry about that.

jeancochrane commented 5 years ago

@hancush Since django-councilmatic now has a v2.5.0 release, I updated this PR to pin django-councilmatic>=2.5 and set the version of this app to 1.0.0. You should now be able to set up this env simply by running pip install -e .[tests].

datamade / django-councilmatic-notifications