[Bug]: sftp OOM on large share indexing

[courville]

Problem description

OOM when scanning large collection on sftp share

Steps to reproduce the issue

index large collection wait and OOM

Expected behavior

No response

Your phone/tablet/androidTV model

nvidia shield TV

Operating system version

Android 9

Application version and app store

No response

Additional system information

No response

Debug logs

W/.courville.nov(14747): Throwing OutOfMemoryError "Failed to allocate a 524304 byte allocation with 481368 free bytes and 470KB until OOM, max allowed footprint 536870912, growth limit 536870912"
--------- beginning of crash
E/AndroidRuntime(14747): FATAL EXCEPTION: ScanWorker
E/AndroidRuntime(14747): Process: org.courville.nova, PID: 14747
E/AndroidRuntime(14747): java.lang.OutOfMemoryError: Failed to allocate a 524304 byte allocation with 481368 free bytes and 470KB until OOM, max allowed footprint 536870912, growth limit 536870912
E/AndroidRuntime(14747):    at com.jcraft.jsch.Channel$MyPipedInputStream.<init>(SourceFile:348)
E/AndroidRuntime(14747):    at com.jcraft.jsch.ChannelSftp.start(SourceFile:230)
E/AndroidRuntime(14747):    at com.jcraft.jsch.Channel.connect(SourceFile:151)
E/AndroidRuntime(14747):    at com.jcraft.jsch.Channel.connect(SourceFile:144)
E/AndroidRuntime(14747):    at com.archos.filecorelibrary.sftp.SFTPSession.getSFTPChannel(SourceFile:52)
E/AndroidRuntime(14747):    at com.archos.filecorelibrary.sftp.SFTPRawLister.getFileList(SourceFile:47)
E/AndroidRuntime(14747):    at com.archos.filecorelibrary.sftp.SFTPRawLister.getFileList(SourceFile:39)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.FileVisitor.recurse(SourceFile:68)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.FileVisitor.recurse(SourceFile:72)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.FileVisitor.recurse(SourceFile:72)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.FileVisitor.visit(SourceFile:53)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.NetworkScannerServiceVideo.doScan(SourceFile:488)
E/AndroidRuntime(14747):    at com.archos.mediaprovider.video.NetworkScannerServiceVideo.handleMessage(SourceFile:314)
E/AndroidRuntime(14747):    at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime(14747):    at android.os.Looper.loop(Looper.java:193)
E/AndroidRuntime(14747):    at android.os.HandlerThread.run(HandlerThread.java:65)

[courville]

Seems to be a regression after https://github.com/nova-video-player/aos-FileCoreLibrary/commit/3eeee79bb8fc7399a1c0d0090f0d1253ec01a544 @phhusson seems that some of our old patches were useful afterall...

[courville]

Confirmed to be a regression.

[courville]

TODO: forked https://github.com/mwiede/jsch here https://github.com/nova-video-player/jsch-mwiede and reapply jcsh-0.1.51-parches.zip and do proper MR. See https://github.com/nova-video-player/aos-AVP/issues/223

[courville]

Perhaps fix is contained in former Archos patches namely:

From 14cfe66c05c30776f53802075d6f7dbf6163984c Mon Sep 17 00:00:00 2001
From: Alexandre Roux <rouxdanzi@archos.com>
Date: Thu, 21 Jul 2016 17:33:48 +0200
Subject: [PATCH 8/9] improving sftp : sometimes session were being
 disconnected even when a channel was still using it. Fix

---
 src/com/jcraft/jsch/ChannelSftp.java | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/com/jcraft/jsch/ChannelSftp.java b/src/com/jcraft/jsch/ChannelSftp.java
index fa2b35d..cdd350d 100644
--- a/src/com/jcraft/jsch/ChannelSftp.java
+++ b/src/com/jcraft/jsch/ChannelSftp.java
@@ -29,8 +29,9 @@ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

 package com.jcraft.jsch;

-import java.io.*;
+import com.archos.filecorelibrary.sftp.SFTPSession;

+import java.io.*;
 import java.util.Vector;

 public class ChannelSftp extends ChannelSession{
@@ -2461,6 +2462,8 @@ public class ChannelSftp extends ChannelSession{
     return true;
   }
   private boolean _sendCLOSE(byte[] handle, Header header) throws Exception{
+    //release session so that it can be disconnected if needed
+    SFTPSession.getInstance().releaseSession(this);
     sendCLOSE(handle);
     return checkStatus(null, header);
   }
--
2.20.1

and

From 4423e3b9d2d79c3f7fd0701833b842a6323dcb15 Mon Sep 17 00:00:00 2001
From: Alexandre Roux <rouxdanzi@archos.com>
Date: Fri, 22 Jul 2016 11:37:49 +0200
Subject: [PATCH 9/9] completing missing channel disconnection + releasesession
 on disconnect

---
 src/com/jcraft/jsch/ChannelSftp.java | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/com/jcraft/jsch/ChannelSftp.java b/src/com/jcraft/jsch/ChannelSftp.java
index cdd350d..787c43d 100644
--- a/src/com/jcraft/jsch/ChannelSftp.java
+++ b/src/com/jcraft/jsch/ChannelSftp.java
@@ -2859,6 +2859,7 @@ public class ChannelSftp extends ChannelSession{

   public void disconnect(){
     super.disconnect();
+    SFTPSession.getInstance().releaseSession(this);
   }

   private boolean isPattern(String path, byte[][] utf8){
--
2.20.1

Note that reintegrating current jsch into nova source code is not that trivial since new jsch requires java16.