Need to paginate overall failure list for a scraper

[kmtracey]

The addresses scraper on staging has over 46K total failures. Attempting to go to the linked page (http://columbusco-staging.openrural.org/data-dashboard/addresses/failures/) gets you (eventually....) a "site maintenance" page. I'm guessing the request is taking too long to process, since we're not doing any pagination on that 46+K list of failures...

[kmtracey]

Or possibly it's related to some other issue...attempting to click on the failure list for any detail run shows Internal Server Error (e.g. http://columbusco-staging.openrural.org/data-dashboard/addresses/165/failures/). Locally I'm getting a NoReverseMatch:

Environment:

Request Method: GET Request URL: http://localhost:8000/data-dashboard/addresses/9/failures/

Django Version: 1.3.1 Python Version: 2.6.6 Installed Applications: ['djcelery', 'openrural', 'gunicorn', 'openrural.data_dashboard', 'django.contrib.messages', 'background_task', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.comments', 'django.contrib.contenttypes', 'django.contrib.gis', 'django.contrib.humanize', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.staticfiles', 'compressor', 'ebdata.blobs', 'ebdata.geotagger', 'ebpub.accounts', 'ebpub.alerts', 'ebpub.db', 'ebpub.geocoder', 'ebpub.moderation', 'ebpub.neighbornews', 'ebpub.openblockapi', 'ebpub.openblockapi.apikey', 'ebpub.petitions', 'ebpub.preferences', 'ebpub.richmaps', 'ebpub.savedplaces', 'ebpub.streets', 'ebpub.widgets', 'obadmin.admin', 'olwidget', 'easy_thumbnails', 'ebdata.nlp', 'ebdata.templatemaker', 'ebdata.textmining', 'ebpub.metros', 'ebpub.utils', 'ebpub.geocoder', 'ebpub.geocoder.parser', 'south'] Installed Middleware: ('django.middleware.gzip.GZipMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'ebpub.accounts.middleware.UserMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.http.ConditionalGetMiddleware')

Traceback: File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1. response = callback(request, _callback_args, *_callback_kwargs) File "/home/kmtracey/projects/openrural/columbus-county-nc/openrural/data_dashboard/views.py" in list_failures
2. return render(request, 'data_dashboard/list_failures.html', context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/shortcuts/init.py" in render
3. return HttpResponse(loader.render_to_string(_args, *_kwargs), File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/loader.py" in render_to_string
4. return t.render(context_instance) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
5. return self._render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in _render
6. return self.nodelist.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
7. bits.append(self.render_node(node, context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render_node
8. return node.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/loader_tags.py" in render
9. return compiled_parent._render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in _render
10. return self.nodelist.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
11. bits.append(self.render_node(node, context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render_node
12. return node.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/loader_tags.py" in render
13. return compiled_parent._render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in _render
14. return self.nodelist.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
15. bits.append(self.render_node(node, context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render_node
16. return node.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/loader_tags.py" in render
17. result = block.nodelist.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
18. bits.append(self.render_node(node, context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render_node
19. return node.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/defaulttags.py" in render
20. nodelist.append(node.render(context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/defaulttags.py" in render
21. return self.nodelist_true.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render
22. bits.append(self.render_node(node, context)) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/base.py" in render_node
23. return node.render(context) File "/home/kmtracey/.virtualenvs/openrural/lib/python2.6/site-packages/django/template/defaulttags.py" in render
24. raise e

Exception Type: NoReverseMatch at /data-dashboard/addresses/9/failures/ Exception Value: Reverse for 'db_newsitem_change' with arguments '('',)' and keyword arguments '{}' not found.

[sarahdooley]

@rebecca-caktus would you give an estimate for this task?

[ghost]

The page was failing (after taking forever to load) because the admin page to which we were linking to "fix" geocoded items doesn't exist for Address items (which aren't geocoded). Therefore, a TemplateSyntaxError was being thrown.