search

Metro-Records / la-metro-dashboard

An Airflow-based dashboard for LA Metro
4 stars 0 forks source link

Debug failing hourly processing #46

Closed hancush closed 3 years ago

hancush commented 3 years ago

compile_pdfs is failing with a mysterious exception:

[2020-07-31 19:10:17,474] {docker.py:258} INFO - Events:   0%|          | 0/304 [00:00<?, ?it/s]
[2020-07-31 19:10:17,517] {docker.py:258} INFO - Events:   0%|          | 0/304 [00:00<?, ?it/s]
[2020-07-31 19:10:17,618] {docker.py:258} INFO - Traceback (most recent call last):
[2020-07-31 19:10:17,619] {docker.py:258} INFO - File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
[2020-07-31 19:10:17,620] {docker.py:258} INFO - return self.get(**kwargs), False
[2020-07-31 19:10:17,620] {docker.py:258} INFO - File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get
[2020-07-31 19:10:17,620] {docker.py:258} INFO - self.model._meta.object_name
[2020-07-31 19:10:17,621] {docker.py:258} INFO - lametro.models.DoesNotExist: EventPacket matching query does not exist.
[2020-07-31 19:10:17,621] {docker.py:258} INFO - During handling of the above exception, another exception occurred:
[2020-07-31 19:10:17,622] {docker.py:258} INFO - Traceback (most recent call last):
[2020-07-31 19:10:17,622] {docker.py:258} INFO - File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 164, in __get__
[2020-07-31 19:10:17,622] {docker.py:258} INFO - rel_obj = self.field.get_cached_value(instance)
[2020-07-31 19:10:17,622] {docker.py:258} INFO - File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/mixins.py", line 13, in get_cached_value
[2020-07-31 19:10:17,623] {docker.py:258} INFO - return instance._state.fields_cache[cache_name]
[2020-07-31 19:10:17,623] {docker.py:258} INFO - KeyError: 'bill'
[2020-07-31 19:10:17,623] {docker.py:258} INFO - During handling of the above exception, another exception occurred:
[2020-07-31 19:10:17,623] {docker.py:258} INFO - Traceback (most recent call last):
[2020-07-31 19:10:17,624] {docker.py:258} INFO - File "manage.py", line 10, in <module>
[2020-07-31 19:10:17,624] {docker.py:258} INFO - execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
[2020-07-31 19:10:17,624] {docker.py:258} INFO - utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
[2020-07-31 19:10:17,624] {docker.py:258} INFO - self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
[2020-07-31 19:10:17,625] {docker.py:258} INFO - self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
[2020-07-31 19:10:17,625] {docker.py:258} INFO - output = self.handle(*args, **options)
  File "/app/lametro/management/commands/compile_pdfs.py", line 43, in handle
[2020-07-31 19:10:17,625] {docker.py:258} INFO - self._compile_events()
  File "/app/lametro/management/commands/compile_pdfs.py", line 51, in _compile_events
[2020-07-31 19:10:17,625] {docker.py:258} INFO - event_packet, created = EventPacket.objects.get_or_create(event=event)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
[2020-07-31 19:10:17,625] {docker.py:258} INFO - return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 541, in get_or_create
[2020-07-31 19:10:17,626] {docker.py:258} INFO - return self._create_object_from_params(kwargs, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 575, in _create_object_from_params
[2020-07-31 19:10:17,626] {docker.py:258} INFO - obj = self.create(**params)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 422, in create
[2020-07-31 19:10:17,626] {docker.py:258} INFO - obj.save(force_insert=True, using=self.db)
  File "/app/lametro/models.py", line 664, in save
[2020-07-31 19:10:17,626] {docker.py:258} INFO - self._merge_docs()
  File "/app/lametro/models.py", line 680, in _merge_docs
[2020-07-31 19:10:17,627] {docker.py:258} INFO - requests.post(merge_url, json=self.related_files)
  File "/app/lametro/models.py", line 743, in related_files
[2020-07-31 19:10:17,627] {docker.py:258} INFO - bill_packet = BillPacket(bill=entity.bill)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 178, in __get__
[2020-07-31 19:10:17,627] {docker.py:258} INFO - rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 145, in get_object
[2020-07-31 19:10:17,628] {docker.py:258} INFO - return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get
[2020-07-31 19:10:17,628] {docker.py:258} INFO - self.model._meta.object_name
lametro.models.DoesNotExist: LAMetroBill matching query does not exist.

Sentry: https://sentry.io/organizations/datamade/issues/1797682110/?project=2131912&referrer=slack

Why is this happening, specifically why is it happening on the staging site? (Opened here, because it's blocking QA.)

hancush commented 3 years ago

I looked back through the last several instances of this error and noticed they all had to do with the same bill, ocd-bill/08460e15-d2e4-41e5-a7a6-2c3be8946a3f. I shelled into the server and confirmed that, while it is not private, it doesn't come through in the LAMetroBill queryset. I applied a patch to handle this exception in https://github.com/datamade/la-metro-councilmatic/pull/637.