RuntimeError: dictionary changed size during iteration

[fortable1999]

Original report by Damien Nozay (Bitbucket: dnozay, GitHub: dnozay).

---

scenario:

• several tabs making the same request.
• some of these tabs hit issue #386

Django==1.6.5
PyMySQL==0.6.2
Whoosh==2.6.0
certifi==14.05.14
django-cors-headers==0.12
django-filter==0.7
django-haystack==2.2.1
djangorestframework==2.4.2
drf-extensions==0.2.5
pysolr==3.2.0
python-memcached==1.53
readline==6.2.4.1
requests==2.4.0
uWSGI==2.1
wsgiref==0.1.2
xunitparser==1.3.0

Environment:

Request Method: GET
Request URL: http://myhostname/api/testruns/search/?age=7&page=5

Django Version: 1.6.5
Python Version: 2.7.7
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'haystack',
 'corsheaders',
 'rest_framework',
 'builds',
 'bugzilla',
 'testruns')
Installed Middleware:
('corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')

Traceback:
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  112.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/rest_framework/viewsets.py" in view
  78.             return self.dispatch(request, *args, **kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  57.         return view_func(*args, **kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  401.             response = self.handle_exception(exc)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  398.             response = handler(request, *args, **kwargs)
File "./testruns/views.py" in search
  125.             items = paginator.page(page)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/django/core/paginator.py" in page
  55.         return self._get_page(self.object_list[bottom:top], number, self)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/query.py" in __getitem__
  266.                 self._fill_cache(start, bound)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/query.py" in _fill_cache
  166.         results = self.query.get_results(**kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/backends/__init__.py" in get_results
  644.                 self.run(**kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/backends/__init__.py" in run
  562.         results = self.backend.search(final_query, **search_kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/backends/__init__.py" in wrapper
  34.             return func(obj, query_string, *args, **kwargs)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py" in search
  457.             results = self._process_results(raw_page, highlight=highlight, query_string=query_string, spelling_query=spelling_query, result_class=result_class)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py" in _process_results
  601.             app_label, model_name = raw_result[DJANGO_CT].split('.')
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/searching.py" in __getitem__
  1501.         if fieldname in self.fields():
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/searching.py" in fields
  1389.             self._fields = self.searcher.stored_fields(self.docnum)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/reading.py" in stored_fields
  1225.         return self.readers[segmentnum].stored_fields(segmentdoc)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/reading.py" in stored_fields
  712.         sfs = self._perdoc.stored_fields(docnum)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/codec/whoosh3.py" in stored_fields
  489.         v = reader[docnum]
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/columns.py" in __getitem__
  1207.             v = self._child[docnum]
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/columns.py" in __getitem__
  807.             v = VarBytesColumn.Reader.__getitem__(self, docnum)
File "/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/util/cache.py" in wrapper
  94.                                           key=itemgetter(1)):
File "/usr/local/lib/python2.7/heapq.py" in nsmallest
  432.     result = _nsmallest(n, it)

Exception Type: RuntimeError at /api/testruns/search/
Exception Value: dictionary changed size during iteration

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

I'm guessing the issue is that you're sharing the same searcher/reader across different threads? Reading isn't multi-threaded; each thread needs its own searcher.

[fortable1999]

Original comment by Damien Nozay (Bitbucket: dnozay, GitHub: dnozay).

---

/opt/nginx/test_dashboard/tools/lib/python2.7/site-packages/whoosh/util/cache.py in wrapper

stats    = [0, 3904]
args     = (<CompressedBytes.Reader>, 335L)
lastused = {(<CompressedBytes.Reader>, 335L): 1410642401.113477,
 (<CompressedBytes.Reader>, 0L): 1410642397.952562
...

/usr/local/lib/python2.7/heapq.py in nsmallest:

Variable    Value
it   = <itertools.izip object at 0x7f7264753dd0>
in2  = <itertools.tee object at 0x7f7275175ef0>
in1  = <itertools.tee object at 0x7f72647eb440>
key  = <operator.itemgetter object at 0x7f72743b4450>
n    = 10
iterable = <dictionary-itemiterator object at 0x7f725c1861b0>