search

flaxsearch / flaxcode

Automatically exported from code.google.com/p/flaxcode
4 stars 1 forks source link

Flax should cope cleanly with running out of disk space #127

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Issue #98 may have been down to this.

Original issue reported on code.google.com by charliej...@gmail.com on 16 Nov 2007 at 4:21

GoogleCodeExporter commented 9 years ago 
This happened to me again and produced an obscure application error. If we can 
come
up with a simple way of checking (even on an irregular basis) and halting 
indexing
and telling the user, that would be good.

Original comment by charliej...@gmail.com on 22 Nov 2007 at 10:36

GoogleCodeExporter commented 9 years ago 
First step is to find a (preferably portable) way of finding out how much free 
space
is available on the volume which a particular directory is mounted on.  Then we 
can
poll it periodically, and warn if it's getting low.  If it can result in memory
errors (due to being unable to swap) though, we're not necessarily going to be 
able
to handle _actually_ running out of disk space cleanly, so we should probably 
set a
minimum value for the amount of disk space available for the indexer to run. 
Fortunately, xapian is quite nicely behaved (give or take about 60k, flush() 
never
uses new disk space, and if there's a failure in flush() the transaction will 
fail to
commit, so the old database will remain valid - new disk space is only used by
add_document() or replace_document()), so if we poll the available disk space 
after
every document (or every few documents if this turns out to have noticeable
performance impact) we should be ok.

Original comment by boulton.rj@gmail.com on 22 Nov 2007 at 10:47

GoogleCodeExporter commented 9 years ago 
On windows at least each volume has it's own disk drive letter? In which case 
the
free space for a directory is the same as the free space of the drive on which 
the
directory lives. This is provided, I think, by the win32api.GetDiskFreeSpaceEx.

Original comment by paul.x.r...@googlemail.com on 22 Nov 2007 at 2:58

GoogleCodeExporter commented 9 years ago 
http://msdn2.microsoft.com/en-us/library/aa364937.aspx

Original comment by paul.x.r...@googlemail.com on 22 Nov 2007 at 3:00

GoogleCodeExporter commented 9 years ago

Original comment by paul.x.r...@googlemail.com on 23 Nov 2007 at 1:24

GoogleCodeExporter commented 9 years ago 
Fixed on windows - I'll add a linux (mac) test too in due course. Unfortunately
there's no cross platform python way of checking as far as I can tell.

Original comment by paul.x.r...@googlemail.com on 25 Nov 2007 at 5:47