search

iiab / internet-in-a-box

Humane Heritage - OLD VERSION
Other
103 stars 16 forks source link

Search on full Wikipedia fails on GoFlex Home: Cannot allocate Memory #84

Closed braddockcg closed 11 years ago

braddockcg commented 11 years ago

An attempt to do a title search on the full edition of Wikipedia fails on the GoFlex Home with 128MB RAM and 1GB of swap turned on. The user receives an "Error loading page" message.

This is the stacktrace from apache on a search for 'kiribati'. This search succeeds on my desktop machine. Seems to fail on any search.

Title search is WORKING for the smaller versions of Wikipedia (like 0.7 and Wikipedia for Schools) on the GoFlex Home.

With 1GB of swap I should think this would work. There is something in there about a mmap.

[Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] mod_wsgi (pid=684): Exception occurred processing WSGI script '/var/www/iiab.wsgi'., referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] Traceback (most recent call last):, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1836, in call, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return self.wsgi_app(environ, start_response), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1820, in wsgi_app, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] response = self.make_response(self.handle_exception(e)), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1403, in handle_exception, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] reraise(exc_type, exc_value, tb), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1817, in wsgi_app, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] response = self.full_dispatch_request(), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1477, in full_dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] rv = self.handle_user_exception(e), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1381, in handle_user_exception, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] reraise(exc_type, exc_value, tb), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1475, in full_dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] rv = self.dispatch_request(), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1461, in dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return self.view_functionsrule.endpoint, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/iiab/zim_views.py", line 84, in search, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109](pagination, suggestion) = paginated_search(index_dir, ["title", "content"], query, page), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/iiab/whoosh_search.py", line 61, in paginated_search, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] with ix.searcher() as searcher:, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/index.py", line 322, in searcher, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return Searcher(self.reader(), fromindex=self, kwargs), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 334, in reader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] info.generation, reuse=reuse), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 315, in _reader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return segreader(segments[0]), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 310, in segreader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] generation=generation), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filereading.py", line 70, in init, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] self.files = OverlayStorage(segment.open_compound_file(storage),, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/codec/base.py", line 656, in open_compound_file, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return CompoundStorage(storage, name), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/compound.py", line 58, in init**, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] self.source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] error: [Errno 12] Cannot allocate memory, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012

jasco commented 11 years ago

Is your desktop a 64 bit system? This might be address space exhaustion if the index is too large for the goflex address space. (While usual solution is mmap windowing, might in this case be easier to search multiple smaller indices.)

-Joel

On Sun, Sep 22, 2013 at 4:17 PM, Braddock Gaskill notifications@github.comwrote:

An attempt to do a title search on the full edition of Wikipedia fails on the GoFlex Home with 128MB RAM and 1GB of swap turned on. The user receives an "Error loading page" message.

This is the stacktrace from apache on a search for 'kiribati'. This search succeeds on my desktop machine. Seems to fail on any search.

Title search is WORKING for the smaller versions of Wikipedia (like 0.7 and Wikipedia for Schools) on the GoFlex Home.

With 1GB of swap I should think this would work. There is something in there about a mmap.

[Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] mod_wsgi (pid=684): Exception occurred processing WSGI script '/var/www/iiab.wsgi'., referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] Traceback (most recent call last):, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1836, in call, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return self.wsgi_app(environ, start_response), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1820, in wsgi_app, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] response = self.make_response(self.handle_exception(e)), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1403, in handle_exception, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] reraise(exc_type, exc_value, tb), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1817, in wsgi_app, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] response = self.full_dispatch_request(), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1477, in full_dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] rv = self.handle_user_exception(e), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1381, in handle_user_exception, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] reraise(exc_type, exc_value, tb), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1475, in full_dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] rv = self.dispatch_request(), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/flask/app.py", line 1461, in dispatch_request, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return self.view_functionsrule.endpoint http://**req.view_args, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/iiab/zim_views.py", line 84, in search, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] client 192.168.1.109http://pagination,%20suggestion= paginated_search(index_dir, ["title", "content"], query, page), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/mnt/data/knowledge/sys/venv/lib/python2.6/site-packages/iiab/whoosh_search.py", line 61, in paginated_search, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] with ix.searcher() as searcher:, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/index.py", line 322, in searcher, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return Searcher(self.reader(), fromindex=self, _kwargs), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 334, in reader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] info.generation, reuse=reuse), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 315, in reader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return segreader(segments[0]), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/fileindex.py", line 310, in segreader, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] generation=generation), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filereading.py", line 70, in init, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] self.files = OverlayStorage(segment.open_compound_file(storage),, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/codec/base.py", line 656, in open_compound_file, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] return CompoundStorage(storage, name), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/compound.py", line 58, in init, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] self.source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ), referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012 [Sun Sep 22 23:13:53 2013] [error] [client 192.168.1.109] error: [Errno 12] Cannot allocate memory, referer: http://know/iiab/zim/iframe/wikipedia_en_all_nopic_01_2012

— Reply to this email directly or view it on GitHubhttps://github.com/braddockcg/internet-in-a-box/issues/84 .

braddockcg commented 11 years ago

Interesting idea, especially since the search index is just a little bigger than 2 GB. so we had the OLPC folks test it on their 32-bit ARM XO-4 server and search there worked on the large wikipedia.

So maybe search needs a block of real RAM for something.

braddockcg commented 11 years ago

I have to revisit Joel's 32-bit theory. I created a 32-bit Fedora 18 virtual machine, and I can recreate the error even when there seems to be a reasonable amount of RAM.

Here is the strace where the error occurs:

18097 mmap2(NULL, 2136145874, PROT_READ, MAP_SHARED, 6, 0) = -1 ENOMEM (Cannot allocate mem ory)

The size 2136145874 is just shy of the signed 32-bit limit of 2147483648, and I assume already allocated memory puts the process over the limit.

So I don't know why on the OLPC XO-4, which is 32-bit ARM, mmap can work. A bit of a mystery.

Unfortunately this is a show stopper for shipping our 6 overdue orders, so I'll keep hacking at it.

braddockcg commented 11 years ago

Here is a cleaner copy of that exception backtrace, for the record:

Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/flask/app.py", line 1687, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python2.7/site-packages/flask/app.py", line 1344, in dispatch_request return self.view_functionsrule.endpoint File "/root/repo/iiab/zim_views.py", line 86, in search (pagination, suggestion) = paginated_search(index_dir, ["title", "content"], query, page) File "/root/repo/iiab/whoosh_search.py", line 61, in paginated_search with ix.searcher() as searcher: File "/usr/lib/python2.7/site-packages/whoosh/index.py", line 322, in searcher return Searcher(self.reader(), fromindex=self, kwargs) File "/usr/lib/python2.7/site-packages/whoosh/filedb/fileindex.py", line 334, in reader info.generation, reuse=reuse) File "/usr/lib/python2.7/site-packages/whoosh/filedb/fileindex.py", line 315, in _reader return segreader(segments[0]) File "/usr/lib/python2.7/site-packages/whoosh/filedb/fileindex.py", line 310, in segreader
generation=generation) File "/usr/lib/python2.7/site-packages/whoosh/filedb/filereading.py", line 70, in init
self.files = OverlayStorage(segment.open_compound_file(storage), File "/usr/lib/python2.7/site-packages/whoosh/codec/base.py", line 656, in open_compound_file return CompoundStorage(storage, name) File "/usr/lib/python2.7/site-packages/whoosh/filedb/compound.py", line 58, in init** self.source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ) error: [Errno 12] Cannot allocate memory

jasco commented 11 years ago

The limit on available virtual space is reduced by system allocated and other program memory space sometimes by a fairly significant amount. There is also the problem of virtual space fragmenting, if I recall properly. When I last encountered this I seem to recall establishing an adaptive reduction of mapped size and then proceeding with windowing. If it's really of interest I could go search for the details of the approach I used last time. However, as I mentioned, if this is the issue, it might be more work to retrofit whoosh than to split the databases.

On Mon, Sep 30, 2013 at 11:47 AM, Braddock Gaskill <notifications@github.com

wrote:

I have to revisit Joel's 32-bit theory. I created a 32-bit Fedora 18 virtual machine, and I can recreate the error even when there seems to be a reasonable amount of RAM.

Here is the strace where the error occurs:

18097 mmap2(NULL, 2136145874, PROT_READ, MAP_SHARED, 6, 0) = -1 ENOMEM (Cannot allocate mem ory)

The size 2136145874 is just shy of the signed 32-bit limit of 2147483648, and I assume already allocated memory puts the process over the limit.

So I don't know why on the OLPC XO-4, which is 32-bit ARM, mmap can work. A bit of a mystery.

Unfortunately this is a show stopper for shipping our 6 overdue orders, so I'll keep hacking at it.

— Reply to this email directly or view it on GitHubhttps://github.com/braddockcg/internet-in-a-box/issues/84#issuecomment-25391798 .

braddockcg commented 11 years ago

Just for the record, the XO-4 and my VM are running an identical version of Whoosh.

yum list installed |grep -i who Whoosh.noarch 2.4.1-1 @xs-iiab

So it isn't a Whoosh versioning thing (there was some mention of mmap in recent whoosh changefile)

braddockcg commented 11 years ago

Joel, I know nothing of whoosh. How do we "split the databases"? Do we have to rebuild them? This is James' department.

braddockcg commented 11 years ago

From the Whoosh author via e-mail:

Matt Chaput
1:57 PM (2 hours ago)

Currently, we get an mmap2 error on 32-bit machines when attempt to search because Whoosh tries to mmap the entire index file, which hits the signed 32-bit address space limit for the process. Our index files are between 2GB and 8GB.

To disable mmap, open the index like this:

from whoosh.filedb import FileStorage

fs = FileStorage("path/to/index", supports_mmap=False) ix = fs.open_index()

Cheers,

Matt