search

cdent / tank

Tiddlers Are N[eo][tw] Knowledge
BSD 3-Clause "New" or "Revised" License
12 stars 7 forks source link

whoosher sometimes fails with pickle error #86

Closed cdent closed 10 years ago

cdent commented 10 years ago

At the moment no more detail than

2014-03-01 19:40:33,850 DEBUG tiddlywebplugins.whoosher: whoosher: exception whi
le indexing: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tiddlywebplugins/whoosher.py", li
ne 363, in _tiddler_change_handler
    writer.commit()
  File "/usr/local/lib/python2.7/dist-packages/whoosh/writing.py", line 935, in 
commit
    self._commit_toc(finalsegments)
  File "/usr/local/lib/python2.7/dist-packages/whoosh/writing.py", line 882, in 
_commit_toc
    toc.write(self.storage, self.indexname)
  File "/usr/local/lib/python2.7/dist-packages/whoosh/index.py", line 693, in wr
ite
    pickle.dumps(field)
  File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

But my guess is that there's something unexpected on the tiddler, perhaps the creator which is a property, that is upsetting pickle. It may also be something coming off the caching store. In any case, the only attributes that are looked at are those described by the schema. Given that it seems mostly likely to be the creator.

It also appears that it only happens on the first creation of a tiddler, which supports the creator theory.

cdent commented 10 years ago

Struggling to replicate this. It may be because of upgrading whoosh to 2.6.0. Not sure.

In any case, the longer term plan is for dual search system (similar to tiddlyspace) so this may be less of an issue.

cdent commented 10 years ago

And of course as soon as I leave that comment the error shows up.

cdent commented 10 years ago

This may have something to do with mod wsgi: https://groups.google.com/forum/#!topic/whoosh/a6rjO-2TTAg and https://bitbucket.org/mchaput/whoosh/issue/314/picklingerror-cant-pickle-apache-windows-7

So for now I've made tank run with its own daemon process to see if that helps.

cdent commented 10 years ago

Hasn't happened since the 6th.