search

palewire / django-calaccess-downloads-website

An open-source archive of campaign finance and lobbying disclosure data from the California Secretary of State’s CAL-ACCESS database
http://calaccess.californiacivicdata.org
MIT License
3 stars 4 forks source link

500 error on /downloads/latest/ #194

Closed simonw closed 2 years ago

simonw commented 3 years ago

https://calaccess.californiacivicdata.org/downloads/latest/ is giving me a Django error page:

IndexError at /downloads/latest/

Copied traceback:

Environment:

Request Method: GET
Request URL: http://calaccess.californiacivicdata.org/downloads/latest/

Django Version: 3.2.7
Python Version: 3.9.7
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.contrib.sites',
 'calaccess_raw',
 'calaccess_scraped',
 'calaccess_processed',
 'calaccess_processed_filings',
 'calaccess_processed_elections',
 'calaccess_processed_flatfiles',
 'opencivicdata.core.apps.BaseConfig',
 'opencivicdata.elections.apps.BaseConfig',
 'calaccess_website',
 'storages',
 'toolbox',
 'whitenoise.runserver_nostatic']
Installed Middleware:
('whitenoise.middleware.WhiteNoiseMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')

Template error:
In template /app/calaccess_website/templates/calaccess_website/version/sections/errata.html, error at line 23
   list index out of range
   13 :           <th class='col-xs-6'>Name</th>
   14 :           <th class="col-xs-3 right">Errors</th>
   15 :           <th class="col-xs-3 right">CSV</th>
   16 :         </tr>
   17 :       </thead>
   18 :       <tbody>
   19 :       {% for file in raw_files_w_errors %}
   20 :       <tr id="footnote-{{ file.file_name|slugify }}">
   21 :         <td class="break"><a href="{% url 'calaccess_file_detail' slug=file.file_name|slugify %}">{{ file.file_name }}</a></td>
   22 :         <td class="right">{{ file.error_count }}</td>
   23 :         <td class="right"><a href=" {% archive_url file.error_log_archive.name app='raw' is_latest=is_latest %} ">Download &raquo;</a></td>
   24 :       </tr>
   25 :       {% endfor %}
   26 :       </tbody>
   27 :       </table>
   28 :     </div>
   29 :   </div>
   30 : </section>
   31 : {% endif %}
   32 : 

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/core/handlers/base.py", line 204, in _get_response
    response = response.render()
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/response.py", line 83, in rendered_content
    return template.render(context, self._request)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/loader_tags.py", line 195, in render
    return template.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 172, in render
    return self._render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 162, in _render
    return self.nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/defaulttags.py", line 312, in render
    return nodelist.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/defaulttags.py", line 211, in render
    nodelist.append(node.render_annotated(context))
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/app/.heroku/python/lib/python3.9/site-packages/django/template/library.py", line 192, in render
    output = self.func(*resolved_args, **resolved_kwargs)
  File "/app/calaccess_website/templatetags/calaccess_website_tags.py", line 40, in archive_url
    return f"https://{current_site.domain}/redirect/latest/{app}/{file_path.split('/')[1]}"

Exception Type: IndexError at /downloads/latest/
Exception Value: list index out of range
simonw commented 3 years ago

Relevant code: https://github.com/california-civic-data-coalition/django-calaccess-downloads-website/blob/62a94cd5935a070916542793afb81b8f14460d6d/calaccess_website/templatetags/calaccess_website_tags.py#L30-L40

file_path here is the empty string.

It looks like that tag is being called here: https://github.com/california-civic-data-coalition/django-calaccess-downloads-website/blob/62a94cd5935a070916542793afb81b8f14460d6d/calaccess_website/templates/calaccess_website/version/sections/errata.html#L23

{% archive_url file.error_log_archive.name app='raw' is_latest=is_latest %}

So file.error_log_archive.name is the empty string here.

palewire commented 3 years ago

Thanks for flagging this. It is byproduct of two recent cost cutting moves. We migrated the database backend from AWS to Heroku, and the static file storage from S3 to the Internet Archive.

In the final weeks of election season, we noticed that our upload speeds to Internet Archive were drastically decreasing, leading to timeouts.

In the interest of generating the core files we needed to the cover the ongoing race, I trimmed down the number of static files being generated and uploaded. I haven't found time yet to go back and try to address the root issue.