Enhanced FSDirectory that allow lock disable via API [LUCENE-84]

[asfimport]

Below is a new version of FSDirectory.java. It is based on 1/30/2003 source.

I have added one new version each of getDirectory(File) and getDirectory (String). They take a new third argument 'boolean useLocks'.

The previous 'private static final boolean DISABLE_LOCKS' has been changed to 'private static boolean _disableLocks' and is initialized to false. I also added a new method 'private boolean locksDisabledByProp' that checks the 'disableLuceneLocks' system property.

Method makeLock now checks the static _disableLocks as the first term in an OR clause, the second of which is a call to locksDisabledByProp. This allows use in an applet that does not have write access to the local filesystem, and when the API is used in that way prevents the query of the system property that is also disallowed in an applet by default (at least in Mozilla/Netscape).

From my applet, I can now invoke:

Searcher searcher = new IndexSearcher( IndexReader.open(FSDirectory.getDirectory(indexFile, false, false)));

and I get an IndexSearcher that will work in the applet with no special permissions other than applet JAR signing.

Obviously, email me (jmethot@bea.com) if you have any questions.

************************* FSDirectory.java *******************************

package org.apache.lucene.store;

/* ====================================================================

• The Apache Software License, Version 1.1 *
• Copyright (c) 2001 The Apache Software Foundation. All rights
• reserved. *
• Redistribution and use in source and binary forms, with or without
• modification, are permitted provided that the following conditions
• are met: *
• 1. Redistributions of source code must retain the above copyright
• notice, this list of conditions and the following disclaimer. *
• 2. Redistributions in binary form must reproduce the above copyright
• notice, this list of conditions and the following disclaimer in
• the documentation and/or other materials provided with the
• distribution. *
• 3. The end-user documentation included with the redistribution,
• if any, must include the following acknowledgment:
• "This product includes software developed by the
• Apache Software Foundation (http://www.apache.org/)."
• Alternately, this acknowledgment may appear in the software itself,
• if and wherever such third-party acknowledgments normally appear. *
• 4. The names "Apache" and "Apache Software Foundation" and
• "Apache Lucene" must not be used to endorse or promote products
• derived from this software without prior written permission. For
• written permission, please contact apache@apache.org. *
• 5. Products derived from this software may not be called "Apache",
• "Apache Lucene", nor may "Apache" appear in their name, without
• prior written permission of the Apache Software Foundation. *
• THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
• WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
• OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
• DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR
• ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
• SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
• LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
• USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
• ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
• OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
• OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
• SUCH DAMAGE.
• ==================================================================== *
• This software consists of voluntary contributions made by many
• individuals on behalf of the Apache Software Foundation. For more
• information on the Apache Software Foundation, please see
• <http://www.apache.org/&gt;. */

import java.io.IOException; import java.io.File; import java.io.RandomAccessFile; import java.util.Hashtable;

import org.apache.lucene.util.Constants;

/**

• Straightforward implementation of {@link Directory} as a directory of files.
• <p>If the system property 'disableLuceneLocks' has the String value of
• "true", lock creation will be disabled. *
• @see Directory
• @author Doug Cutting */

public final class FSDirectory extends Directory { /** This cache of directories ensures that there is a unique Directory

• instance per path, so that synchronization on the Directory can be used to
• synchronize access between readers and writers. *
• This should be a WeakHashMap, so that entries can be GC'd, but that would

• require Java 1.2. Instead we use refcounts... */ private static final Hashtable DIRECTORIES = new Hashtable();

  private static boolean _disableLocks = false;

  private boolean locksDisabledByProp() { return Boolean.getBoolean("disableLuceneLocks") || Constants.JAVA_1_1; }

  /** Returns the directory instance for the named location, with option of

• disabled locking. *
• <p>Directories are cached, so that, for a given canonical path, the same
• FSDirectory instance will always be returned. This permits
• synchronization on directories. *
• @param path the path to the directory.
• @param create if true, create, or erase any existing contents.
• @param useLocks if false, don't use locks during index reads. Useful in
• read-only filesystems or from applets.

• @return the FSDirectory for the named file. */ public static FSDirectory getDirectory(String path, boolean create, boolean useLocks) throws IOException { _disableLocks = !useLocks; return getDirectory(new File(path), create); }

  /** Returns the directory instance for the named location. *

• <p>Directories are cached, so that, for a given canonical path, the same
• FSDirectory instance will always be returned. This permits
• synchronization on directories. *
• @param path the path to the directory.
• @param create if true, create, or erase any existing contents.

• @return the FSDirectory for the named file. */ public static FSDirectory getDirectory(String path, boolean create) throws IOException { return getDirectory(new File(path), create); }

  /** Returns the directory instance for the named location, with option of

• disabled locking. *
• <p>Directories are cached, so that, for a given canonical path, the same
• FSDirectory instance will always be returned. This permits
• synchronization on directories. *
• @param file the path to the directory.
• @param create if true, create, or erase any existing contents.
• @param useLocks if false, don't use locks during index reads. Useful in
• read-only filesystems or from applets.

• @return the FSDirectory for the named file. */ public static FSDirectory getDirectory(File file, boolean create, boolean useLocks) throws IOException { _disableLocks = !useLocks; return getDirectory(file, create); }

  /** Returns the directory instance for the named location. *

• <p>Directories are cached, so that, for a given canonical path, the same
• FSDirectory instance will always be returned. This permits
• synchronization on directories. *
• @param file the path to the directory.
• @param create if true, create, or erase any existing contents.

• @return the FSDirectory for the named file. */ public static FSDirectory getDirectory(File file, boolean create) throws IOException { file = new File(file.getCanonicalPath()); FSDirectory dir; synchronized (DIRECTORIES) { dir = (FSDirectory)DIRECTORIES.get(file); if (dir == null) { dir = new FSDirectory(file, create); DIRECTORIES.put(file, dir); } else if (create) { dir.create(); } } synchronized (dir) { dir.refCount++; } return dir; }

  private File directory = null; private int refCount;

  private FSDirectory(File path, boolean create) throws IOException { directory = path;

  if (create) create();

  if (!directory.isDirectory()) throw new IOException(path + " not a directory"); }

  private synchronized void create() throws IOException { if (!directory.exists()) if (!directory.mkdir()) throw new IOException("Cannot create directory: " + directory);

  String[] files = directory.list(); // clear old files for (int i = 0; i <files.length; i++) { File file = new File(directory, files[i]); if (!file.delete()) throw new IOException("couldn't delete " + files[i]); } }

  /* Returns an array of strings, one for each file in the directory. / public final String[] list() throws IOException { return directory.list(); }

  /* Returns true iff a file with the given name exists. / public final boolean fileExists(String name) throws IOException { File file = new File(directory, name); return file.exists(); }

  /* Returns the time the named file was last modified. / public final long fileModified(String name) throws IOException { File file = new File(directory, name); return file.lastModified(); }

  /* Returns the time the named file was last modified. / public static final long fileModified(File directory, String name) throws IOException { File file = new File(directory, name); return file.lastModified(); }

  /* Set the modified time of an existing file to now. / public void touchFile(String name) throws IOException { File file = new File(directory, name); file.setLastModified(System.currentTimeMillis()); }

  /* Returns the length in bytes of a file in the directory. / public final long fileLength(String name) throws IOException { File file = new File(directory, name); return file.length(); }

  /* Removes an existing file in the directory. / public final void deleteFile(String name) throws IOException { File file = new File(directory, name); if (!file.delete()) throw new IOException("couldn't delete " + name); }

  /* Renames an existing file in the directory. / public final synchronized void renameFile(String from, String to) throws IOException { File old = new File(directory, from); File nu = new File(directory, to);

  / This is not atomic. If the program crashes between the call to delete() and the call to renameTo() then we're screwed, but I've been unable to figure out how else to do this... /

  if (nu.exists()) if (!nu.delete()) throw new IOException("couldn't delete " + to);

  if (!old.renameTo(nu)) throw new IOException("couldn't rename " + from + " to " + to); }

  /* Creates a new, empty file in the directory with the given name. Returns a stream writing this file. / public final OutputStream createFile(String name) throws IOException { return new FSOutputStream(new File(directory, name)); }

  /* Returns a stream reading an existing file. / public final InputStream openFile(String name) throws IOException { return new FSInputStream(new File(directory, name)); }

  /** Constructs a {@link Lock} with the specified name. Locks are implemented

  • with {@link File#createNewFile() }.
  • <p>In JDK 1.1 or if system property <I>disableLuceneLocks</I> is the
• string "true", locks are disabled. Assigning this property any other
• string will <B>not</B> prevent creation of lock files. Locks are also
• disabled if getDirectory was invoked with the optional third argument
• useLocks set to false.
• Disabling locks is useful when using Lucene on read-only medium, such
• as CD-ROM, or from an applet that does not have filesystem write
• permission. *
• @param name the name of the lock file

• @return an instance of <code>Lock</code> holding the lock */ public final Lock makeLock(String name) { final File lockFile = new File(directory, name); if (_disableLocks || locksDisabledByProp()) { return new Lock() { public boolean obtain() throws IOException { return true; } public void release() { return; } public String toString() { return "Lock@locksDisabled"; } }; } else { return new Lock() { public boolean obtain() throws IOException { return lockFile.createNewFile(); } public void release() { lockFile.delete(); } public String toString() { return "Lock@" + lockFile; } }; } }

  /** Closes the store to future operations. */ public final synchronized void close() throws IOException { if (--refCount <= 0) { synchronized (DIRECTORIES) { DIRECTORIES.remove(directory); } } }

  /** For debug output. */ public String toString() { return "FSDirectory@" + directory; } }

final class FSInputStream extends InputStream { private class Descriptor extends RandomAccessFile { public long position; public Descriptor(File file, String mode) throws IOException { super(file, mode); } }

Descriptor file = null; boolean isClone;

public FSInputStream(File path) throws IOException { file = new Descriptor(path, "r"); length = file.length(); }

/** InputStream methods */ protected final void readInternal(byte[] b, int offset, int len) throws IOException { synchronized (file) { long position = getFilePointer(); if (position != file.position) { file.seek(position); file.position = position; } int total = 0; do { int i = file.read(b, offset+total, len-total); if (i == -1) throw new IOException("read past EOF"); file.position += i; total += i; } while (total < len); } }

public final void close() throws IOException { if (!isClone) file.close(); }

/** Random-access methods */ protected final void seekInternal(long position) throws IOException { }

protected final void finalize() throws IOException { close(); // close the file }

public Object clone() { FSInputStream clone = (FSInputStream)super.clone(); clone.isClone = true; return clone; } }

final class FSOutputStream extends OutputStream { RandomAccessFile file = null;

public FSOutputStream(File path) throws IOException { file = new RandomAccessFile(path, "rw"); }

/** output methods: */ public final void flushBuffer(byte[] b, int size) throws IOException { file.write(b, 0, size); } public final void close() throws IOException { super.close(); file.close(); }

/** Random-access methods */ public final void seek(long pos) throws IOException { super.seek(pos); file.seek(pos); } public final long length() throws IOException { return file.length(); }

protected final void finalize() throws IOException { file.close(); // close the file }

}

---

Migrated from LUCENE-84 by John Methot, resolved Sep 27 2005 Environment:

Operating System: All
Platform: PC

Attachments: ASF.LICENSE.NOT.GRANTED--FSDirectory.java (versions: 2)

[asfimport]

John Methot (migrated from JIRA)

Created an attachment (id=4743) Proposed new code FSDirectory.java

[asfimport]

Otis Gospodnetic (migrated from JIRA)

John - just so I understand - your change does not add any new functionality, it simply allows you to disable locks by using one of your 2 new methods. Is this because you cannot set the system properly when writing an applet? Couldn't you just call System.setProperty(String key, String value) to set disableLuceneLocks property to true before calling FSDirectory and achieve the same thing?

[asfimport]

Otis Gospodnetic (migrated from JIRA)

The diff is a bit hard to read, contains some formatting changes in addition to changes to the way locking is disabled, but I think it looks good, it compiles, and it doesn't break any unit tests. This is also a modification made to one of the older versions of FSDirectory, but that should be easy to handle.

Anyhow, I'm checking this in, unless somebody complains by tomorrow.

[asfimport]

Otis Gospodnetic (migrated from JIRA)

I do not remember if the attached code was ever committed, but I believe we've had the have support to disable locking for a while now. I'm closing this issue for now.

[asfimport]

John Methot (migrated from JIRA)

Otis, I never saw your comments in the bug, thus my failure to reply. I'm not clear whether your comment in the bug today means you're removing my changes from the codebase (your previous comment says you checked it in).

Vis a vis your prior comment:

"John - just so I understand - your change does not add any new functionality..."

One can't call System.setProperty() from an applet unless the applet has specific permission to do so, as configured on the end-user's machine. The only way to disable locking wihtout special permission, to my knowledge, is via the API.

[asfimport]

John Methot (migrated from JIRA)

Created an attachment (id=10558) New version based on 1.3-final source - easier diffs

[asfimport]

John Methot (migrated from JIRA)

Well, I completely misunderstood Otis' comments that meant he hadn't checked in the change. I didn't realize it hadn't been checked in until I updated to 1.3 and the fix wasn't there.

So here it is again. I've rewritten the changes against 1.3-final source, so the diffs should be a lot more intelligible.

[asfimport]

Otis Gospodnetic (@otisg) (migrated from JIRA)

It looks like this old issue was fixed at some point, as FSDirectory now has the following method:

public static void setDisableLocks(boolean doDisableLocks) { ...