[BUG] java.io.FilePermission error while querying remote restored index with scripted field.

[Dileep-Dora]

Describe the bug

We're trying out searchable snapshots. we've taken snapshot and restored as remote searchable index on search nodes. also we've a scripted field defined.

When querying getting the below exception.

java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
java.base/java.lang.SecurityManager.checkDelete(SecurityManager.java:875)
java.base/sun.nio.fs.UnixPath.checkDelete(UnixPath.java:794)
java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:226)
java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$0(FileCacheFactory.java:58)
org.opensearch.ExceptionsHelper.catchAsRuntimeException(ExceptionsHelper.java:352)
org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$1(FileCacheFactory.java:58)
org.opensearch.index.store.remote.utils.cache.LRUCache.evict(LRUCache.java:363)
org.opensearch.index.store.remote.utils.cache.LRUCache.replaceNode(LRUCache.java:330)
org.opensearch.index.store.remote.utils.cache.LRUCache.compute(LRUCache.java:168)
org.opensearch.index.store.remote.utils.cache.SegmentedCache.compute(SegmentedCache.java:96)
org.opensearch.index.store.remote.filecache.FileCache.compute(FileCache.java:91)
org.opensearch.index.store.remote.utils.TransferManager.fetchBlob(TransferManager.java:59)
org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.fetchBlock(OnDemandBlockSnapshotIndexInput.java:148)
org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.demandBlock(OnDemandBlockIndexInput.java:340)
org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.seekInternal(OnDemandBlockIndexInput.java:311)
org.opensearch.index.store.remote.file.OnDemandBlockIndexInput.readInt(OnDemandBlockIndexInput.java:251)
org.opensearch.index.store.remote.file.OnDemandBlockSnapshotIndexInput.readInt(OnDemandBlockSnapshotIndexInput.java:28)
org.apache.lucene.codecs.lucene90.IndexedDISI.advanceBlock(IndexedDISI.java:474)
org.apache.lucene.codecs.lucene90.IndexedDISI.advanceExact(IndexedDISI.java:459)
org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$21.advanceExact(Lucene90DocValuesProducer.java:904)
org.apache.lucene.index.SingletonSortedSetDocValues.advanceExact(SingletonSortedSetDocValues.java:85)
org.opensearch.index.fielddata.FieldData$12.advanceExact(FieldData.java:434)
org.opensearch.index.fielddata.ScriptDocValues$BinaryScriptDocValues.setNextDocId(ScriptDocValues.java:508)
org.opensearch.index.fielddata.ScriptDocValues$Strings.setNextDocId(ScriptDocValues.java:547)
org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:103)
org.opensearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:56)
if (doc.containsKey('tp') && !doc['tp'].empty ) {
  def 
                                  ^---- HERE

access denied ("java.io.FilePermission" "/data/opensearch/data/nodes/0/cache/tIZdR9UlQZqkFyOsaAIycw/0/RemoteLocalStore/_259_Lucene90_0.dvd.15" "delete")

but the required file permissions are there 0755

Related component

Search:Searchable Snapshots

To Reproduce

1. Index some data
2. Take snapshot
3. restore snapshot as remote
4. query with a scripted field defined

Expected behavior

It should not throw any exception and should work like regular search i.e scripted field on a normal index vs remote index(restored from snapshot as remote).

Additional Details

Opensearch version: 2.13

[andrross]

We ran into a similar problem before where if a script trigged a remote download, then we must elevate privileges for the security manager: https://github.com/opensearch-project/OpenSearch/blob/3125b948029609f354d3153f8ca6391638daefc7/server/src/main/java/org/opensearch/index/store/remote/utils/TransferManager.java#L85

This appears to be the case where even before attempting a remote download we do an eviction from the cache, which results in deleting a local file. That part of the code does not elevate privileges:

https://github.com/opensearch-project/OpenSearch/blob/3125b948029609f354d3153f8ca6391638daefc7/server/src/main/java/org/opensearch/index/store/remote/utils/TransferManager.java#L59

I suspect the fix here is to move the AccessController.doPrivileged call higher up in the call stack to cover this case.

[finnegancarroll]

Thanks for the insight @andrross, i'll test this fix.