Question: How come I can't run on Android itself ?

[AndroidDeveloperLB]

I tried to make a sample app to check it out.

To do this, I granted the storage permissions, and ran this:

        thread {
            val inputFile = File("/storage/emulated/0/test.apkm")
            val outputFile = File("/storage/emulated/0/test.zip")
            UnApkm(inputFile.absolutePath, outputFile.absolutePath)
        }

For some reason, while debugging, I noticed it crashed on this line:

            NativeLong memLimit = new NativeLong(thirdLong);

and the logs:

 FATAL EXCEPTION: Thread-2
    Process: com.lb.apkmtest, PID: 20358
    java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/android-aarch64/libjnidispatch.so) not found in resource path (.)
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
        at com.sun.jna.Native.<clinit>(Native.java:195)
        at com.sun.jna.NativeLong.<clinit>(NativeLong.java:35)
        at org.example.UnApkm.<init>(UnApkm.java:70)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:43)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:14)
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

Attached project here:

ApkmTest using dependency.zip

So I tried to copy the entire file into the project, including its dependencies, but then it crashed on this line:

            LazySodiumJava lazySodium = new LazySodiumJava(new SodiumJava());

And the logs:

FATAL EXCEPTION: Thread-2
    Process: com.lb.apkmtest, PID: 19869
    java.lang.NoClassDefFoundError: com.sun.jna.Native
        at com.goterl.lazycode.lazysodium.utils.LibraryLoader.getSodiumPathInResources(LibraryLoader.java:122)
        at com.goterl.lazycode.lazysodium.utils.LibraryLoader.loadBundledLibrary(LibraryLoader.java:114)
        at com.goterl.lazycode.lazysodium.utils.LibraryLoader.loadLibrary(LibraryLoader.java:84)
        at com.goterl.lazycode.lazysodium.SodiumJava.<init>(SodiumJava.java:34)
        at com.goterl.lazycode.lazysodium.SodiumJava.<init>(SodiumJava.java:23)
        at com.lb.apkmtest.UnApkm.<init>(UnApkm.java:81)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:42)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:13)
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)
     Caused by: java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/android-aarch64/libjnidispatch.so) not found in resource path (.)
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
        at com.sun.jna.Native.<clinit>(Native.java:195)
        at com.sun.jna.Native.register(Native.java:1721)
        at co.libly.resourceloader.SharedLibraryLoader.registerLibraryWithClasses(SharedLibraryLoader.java:74)
        at co.libly.resourceloader.SharedLibraryLoader.loadSystemLibrary(SharedLibraryLoader.java:42)
        at com.goterl.lazycode.lazysodium.utils.LibraryLoader.loadSystemLibrary(LibraryLoader.java:99)
        at com.goterl.lazycode.lazysodium.utils.LibraryLoader.loadLibrary(LibraryLoader.java:81)
        at com.goterl.lazycode.lazysodium.SodiumJava.<init>(SodiumJava.java:34) 
        at com.goterl.lazycode.lazysodium.SodiumJava.<init>(SodiumJava.java:23) 
        at com.lb.apkmtest.UnApkm.<init>(UnApkm.java:81) 
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:42) 
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:13) 
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30) 

Attached here this project:

ApkmTest.zip

So why does it occur? Any way to run it on Android itself?

[souramoo]

Yep! It should run on android too, but as you've noticed, the LazySodiumJava library isn't designed for Android - this is because it is simply a wrapper around libsodium that is actually written in C.

Fortunately there's an easy drop-in replacement as a fix - https://github.com/terl/lazysodium-android ;) So you'll need to change the dependency and possibly the class name, to LazySodiumAndroid but the functions should remain the same!

[AndroidDeveloperLB]

You probably mean this line:

LazySodiumJava lazySodium = new LazySodiumJava(new SodiumJava());

I changed it to this:

            LazySodiumAndroid lazySodium = new LazySodiumAndroid(new SodiumAndroid());

Also had to changed the dependencies a bit. Attached here the output project:

ApkmTest.zip

Seems to work fine, but for some reason the output zip couldn't be opened via a file manager I use (Total Commander), yet it worked fine when I opened it via the PC. Any idea why?

Also, it seems that before the loop of reading, it takes quite some time to initialize. Is the decrypting/decoding so intense?

On a relatively low end device (Xiaomi Redmi 8), it took 4 seconds in total to handle a 23MB file as input (has 18 files inside), and almost all of it was the part before the loop. I guess it can be worse for larger APKM files. Is there a way to optimize it? It's a huge time compared to a simple ZIP file.

Maybe you should offer a listener for the status of the reading, to allow to show progress and report errors.

[souramoo]

Yep, that line sounds about right :)

The output zip isn't a proper zip file as it's missing the end-of-central-directory signature, but this is simply what was encoded into the apkm file pre-crypto. It is still supported by ZipInputStream however so you should still be able to parse it with java or many of the other options

The 4 second delay is likely the hash generation function (the "memory-hard, CPU-intensive hash function" described at https://libsodium.gitbook.io/doc/password_hashing/default_phf ) - but thankfully all this only needs to be done once per apkm file (and alas is necessary) :) It shouldn't be any worse on larger APKM files because it's only the small number of bytes at the very start of the file that are put through this hash function to derive the secret key, not the whole file.

[AndroidDeveloperLB]

1. Is there a way to fix the output file so that file manager apps would be able to open it? Using ZipFile or ZipInputStream , it works fine for me, so I don't get why Total Commander app couldn't handle it:

            try {
                Log.d("AppLog", "opening using ZipInputStream:")
                ZipInputStream(FileInputStream(outputFile)).use {
                    while(true){
                        val entry = it.nextEntry?:return@use
                        Log.d("AppLog", "${entry.name} - ${entry.size}")
                    }
                }
                Log.d("AppLog", "opening using ZipFile:")
                val zipFile = ZipFile(outputFile)
                for (entry in zipFile.entries()) {
                    Log.d("AppLog", "${entry.name} - ${entry.size}")
                }
            }
            catch (e:Exception){
            }

2. As for the 4 seconds, is there a way to minimize it?

3. Is there a way to have this 4 seconds init only once per APKM file? For example, suppose I check some content of the stream, do something else, and then I want to handle the stream again from beginning. Maybe there is a way to cache what's created in this 4 seconds, to avoid having another 4 seconds? If it's possible, I think you should change the static functions so that it would be a part of a normal class, that has the caching per file. The CTOR will be minimal, of course, and won't require any handling of files there.

[souramoo]

1. Yep you can feed the ZipInputStream into a ZipOutputStream and use that to save to a file which should fix this error

2 and 3. You should be able to minimise it using the convenience header functions I've added in - you can call processHeader(i, lazySodium, true) to get your header the first time round, and then next time you open the stream you can just use processHeader(i, lazySodium, false) to skip past the header without doing the crypto stuff and use the previously cached header to process it in decryptStream(i, h, lazySodium); - this header is file specific so beware

[AndroidDeveloperLB]

1. OK I tried this:

UnApkm.decryptStream(FileInputStream(inputFile)).copyTo(ZipOutputStream(FileOutputStream(outputFile)))

This causes this exception:

2020-03-27 14:21:05.356 7434-7497/com.lb.apkmtest E/AndroidRuntime: FATAL EXCEPTION: Thread-2
    Process: com.lb.apkmtest, PID: 7434
    java.util.zip.ZipException: no current ZIP entry
        at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:328)
        at kotlin.io.ByteStreamsKt.copyTo(IOStreams.kt:108)
        at kotlin.io.ByteStreamsKt.copyTo$default(IOStreams.kt:103)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:51)
        at com.lb.apkmtest.MainActivity$onGotPermissions$1.invoke(MainActivity.kt:19)
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

I think the function already created a zip content, so I think it doesn't make sense to provide it here.

So I used this instead:

UnApkm.decryptStream(FileInputStream(inputFile)).copyTo(FileOutputStream(outputFile))

But then, again, the file can't be opened there.

Attached project here: ApkmTest.zip

2.3. I don't think I did it well, or you. Something is weird here. This is what I did:

            Log.d("AppLog", "getting header")
            val lazySodiumAndroid = LazySodiumAndroid(SodiumAndroid())
            val header = UnApkm.processHeader(FileInputStream(inputFile), lazySodiumAndroid, true)
            Log.d("AppLog", "done getting header. handling content")
            UnApkm.processHeader(FileInputStream(inputFile), lazySodiumAndroid, false)
            UnApkm.decryptStream(FileInputStream(inputFile), header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
            Log.d("AppLog", "done handling content. Handle it again:")
            outputFile.delete()
            UnApkm.processHeader(FileInputStream(inputFile), lazySodiumAndroid, false)
            UnApkm.decryptStream(FileInputStream(inputFile), header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
            Log.d("AppLog", "done handling content again")

The exception I get:

2020-03-27 14:37:24.907 9151-9209/com.lb.apkmtest W/System.err: java.io.IOException: decrypto error
2020-03-27 14:37:24.907 9151-9209/com.lb.apkmtest W/System.err:     at com.lb.apkmtest.UnApkm.lambda$decryptStream$0(UnApkm.java:145)
2020-03-27 14:37:24.908 9151-9209/com.lb.apkmtest W/System.err:     at com.lb.apkmtest.-$$Lambda$UnApkm$SQ8gROLqltIDjnsz2Wg8Bb946lk.run(Unknown Source:8)
2020-03-27 14:37:24.908 9151-9209/com.lb.apkmtest W/System.err:     at java.lang.Thread.run(Thread.java:764)
2020-03-27 14:37:24.908 9151-9204/com.lb.apkmtest D/AppLog: done handling content. Handle it again:
2020-03-27 14:37:24.919 9151-9210/com.lb.apkmtest W/System.err: java.io.IOException: decrypto error
    .
2020-03-27 14:37:24.919 9151-9210/com.lb.apkmtest W/System.err:     at com.lb.apkmtest.UnApkm.lambda$decryptStream$0(UnApkm.java:145)
2020-03-27 14:37:24.919 9151-9210/com.lb.apkmtest W/System.err:     at com.lb.apkmtest.-$$Lambda$UnApkm$SQ8gROLqltIDjnsz2Wg8Bb946lk.run(Unknown Source:8)
2020-03-27 14:37:24.919 9151-9210/com.lb.apkmtest W/System.err:     at java.lang.Thread.run(Thread.java:764)

Attached project : ApkmTest.zip

I really suggest to decouple the implementation of LazySodium, so that it could be used on Android too, easily.

[souramoo]

You are creating too many FileInputStreams - the crypted data is stored after the header, and by creating a new stream you are trying to decrypt the header instead of the ciphertext. Processing the header without computation on the same stream skips ahead to the ciphertext.

 Log.d("AppLog", "getting header")
            val lazySodiumAndroid = LazySodiumAndroid(SodiumAndroid())
FileInputStream fis = FileInputStream(inputFile);
            val header = UnApkm.processHeader(fis, lazySodiumAndroid, true)
            Log.d("AppLog", "done getting header. handling content")
            UnApkm.decryptStream(fis, header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
            Log.d("AppLog", "done handling content. Handle it again:")
            outputFile.delete()
fis.close();

fis = FileInputStream(inputFile);
            UnApkm.processHeader(fis, lazySodiumAndroid, false)
            UnApkm.decryptStream(fis, header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
fis.close();
            Log.d("AppLog", "done handling content again")

[AndroidDeveloperLB]

Oh right I forgot to close. I wanted just the header in the beginning, to show that this one takes most of the time, and that later it's almost nothing, so I need to use a new InputStream 3 times : header, content, and again content.

Here's the new code for this:

            Log.d("AppLog", "getting header")
            val lazySodiumAndroid = LazySodiumAndroid(SodiumAndroid())
            val header = FileInputStream(inputFile).use { UnApkm.processHeader(it, lazySodiumAndroid, true) }
            Log.d("AppLog", "done getting header. handling content")
            FileInputStream(inputFile).use {
                UnApkm.processHeader(it, lazySodiumAndroid, false)
                UnApkm.decryptStream(it, header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
            }
            Log.d("AppLog", "done handling content. Handle it again:")
            outputFile.delete()
            FileInputStream(inputFile).use {
                UnApkm.processHeader(it, lazySodiumAndroid, false)
                UnApkm.decryptStream(it, header, lazySodiumAndroid).copyTo(FileOutputStream(outputFile))
            }
            Log.d("AppLog", "done handling content again")

This works fine, but it shows in the logs that each of those took a long time:

ApkmTest.zip

Also, BTW, the file manager still can't handle the output zip file.

[souramoo]

I've fixed the ZIP output bug in the latest commit and release.

How weird! Please do look into if you can see any other bottlenecks - you can always cache the outputstream into a byte array to keep reusing it so you only need to decrypt it once!

But yes crypto operations are cpu/mem heavy.

[AndroidDeveloperLB]

Zip file still can't be opened (via Total Commander app), and the time to handle the content after getting the header is still about as long as getting the header. However, for some reason, after the first handling of the content, the second one takes a short time.

ApkmTest.zip

Saving files content into the RAM is very efficient but very problematic in case the content is too large. I also don't remember if ByteArray on Android has come into the native memory instead of heap. I know Bitmap data has, but I don't remember if ByteArray has. If it's part of the heap, it's even more problematic, because heap size might be too small.

In any case, I suggest not to do it in the library, or if you insist: to provide it as a parameter of how much memory is allowed for it.