search

ankidroid / Anki-Android

AnkiDroid: Anki flashcards on Android. Your secret trick to achieve superhuman information retention.
GNU General Public License v3.0
8.48k stars 2.2k forks source link

[Bug] Unit Tests: java.lang.UnsatisfiedLinkError: 'byte[][] net.ankiweb.rsdroid.NativeMethods.openBackend(byte[])' when mixing `RunWith(AndroidJUnit4::class)` #14796

Open david-allison opened 10 months ago

david-allison commented 10 months ago

A mix of JvmTest with and without the AndroidJUnit4 annotation produces flaky tests

Steps

Apply this patch & run tests on com.ichi2.anki.libanki ```patch Subject: [PATCH] speed --- Index: AnkiDroid/src/test/java/com/ichi2/libanki/FlagTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/FlagTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/FlagTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/FlagTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/FlagTest.kt (date 1700607248378) @@ -21,7 +21,6 @@ import org.junit.Test import org.junit.runner.RunWith -@RunWith(AndroidJUnit4::class) class FlagTest : JvmTest() { /***************** ** Flags * Index: AnkiDroid/src/test/java/com/ichi2/libanki/AbstractSchedTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/AbstractSchedTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/AbstractSchedTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/AbstractSchedTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/AbstractSchedTest.kt (date 1700607248355) @@ -15,7 +15,6 @@ */ package com.ichi2.libanki -import androidx.test.ext.junit.runners.AndroidJUnit4 import com.ichi2.libanki.sched.Counts import com.ichi2.testutils.JvmTest import com.ichi2.utils.KotlinCleanup @@ -23,7 +22,6 @@ import org.hamcrest.Matchers.* import org.json.JSONArray import org.junit.Test -import org.junit.runner.RunWith import kotlin.test.assertEquals import kotlin.test.assertNotNull @@ -32,7 +30,6 @@ @KotlinCleanup("is -> equalTo") @KotlinCleanup("reduce newlines in asserts") @KotlinCleanup("improve increaseAndAssertNewCountsIs") -@RunWith(AndroidJUnit4::class) class AbstractSchedTest : JvmTest() { @Test fun ensureUndoCorrectCounts() { Index: AnkiDroid/build.gradle IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/build.gradle b/AnkiDroid/build.gradle --- a/AnkiDroid/build.gradle (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/build.gradle (date 1700607248350) @@ -388,6 +388,7 @@ testImplementation("androidx.fragment:fragment-testing:$fragments_version") // in a JvmTest we need org.json.JSONObject to not be mocked testImplementation 'org.json:json:20220924' + testImplementation 'io.github.ivanshafran:shared-preferences-mock:1.2.4' // May need a resolution strategy for support libs to our versions androidTestImplementation "androidx.test.espresso:espresso-core:$espresso_version" androidTestImplementation("androidx.test.espresso:espresso-contrib:$espresso_version") { Index: AnkiDroid/src/test/java/com/ichi2/testutils/JvmTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/testutils/JvmTest.kt b/AnkiDroid/src/test/java/com/ichi2/testutils/JvmTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/testutils/JvmTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/testutils/JvmTest.kt (date 1700607248414) @@ -18,6 +18,10 @@ import android.annotation.SuppressLint import androidx.annotation.CallSuper +import androidx.core.content.edit +import com.github.ivanshafran.sharedpreferencesmock.SPMockBuilder +import com.ichi2.anki.AnkiDroidApp +import com.ichi2.anki.CollectionHelper import com.ichi2.anki.CollectionManager import com.ichi2.libanki.ChangeManager import com.ichi2.libanki.Collection @@ -38,6 +42,7 @@ import org.junit.Before import timber.log.Timber import timber.log.Timber.Forest.plant +import java.nio.file.Files open class JvmTest { private fun maybeSetupBackend() { @@ -57,6 +62,12 @@ @Before @CallSuper open fun setUp() { + AnkiDroidApp.sharedPreferencesTestingOverride = SPMockBuilder().createSharedPreferences() + AnkiDroidApp.sharedPrefs().edit { + putString(CollectionHelper.PREF_COLLECTION_PATH, Files.createTempDirectory( + "AnkiDroid-JvmTest" + ).toFile().path) + } TimeManager.resetWith(MockTime(2020, 7, 7, 7, 0, 0, 0, 10)) ChangeManager.clearSubscribers() @@ -83,6 +94,7 @@ @After @CallSuper open fun tearDown() { + AnkiDroidApp.sharedPreferencesTestingOverride = null try { // If you don't tear down the database you'll get unexpected IllegalStateExceptions related to connections col_?.close() Index: AnkiDroid/src/test/java/com/ichi2/libanki/DecksTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/DecksTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/DecksTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/DecksTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/DecksTest.kt (date 1700607248370) @@ -30,7 +30,6 @@ import kotlin.test.assertFalse import kotlin.test.assertTrue -@RunWith(AndroidJUnit4::class) class DecksTest : JvmTest() { @Test fun test_remove() { Index: AnkiDroid/src/test/java/com/ichi2/libanki/PythonExtensionsTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/PythonExtensionsTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/PythonExtensionsTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/PythonExtensionsTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/PythonExtensionsTest.kt (date 1700607248400) @@ -27,7 +27,6 @@ import org.junit.Test import org.junit.runner.RunWith -@RunWith(AndroidJUnit4::class) class PythonExtensionsTest { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/ModelTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/ModelTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/ModelTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/ModelTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/ModelTest.kt (date 1700607248392) @@ -37,7 +37,6 @@ return " data-cloze=\"${data}\"" } -@RunWith(AndroidJUnit4::class) @KotlinCleanup("improve kotlin code where possible") class NotetypeTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/UtilsTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/UtilsTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/UtilsTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/UtilsTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/UtilsTest.kt (date 1700607248411) @@ -23,7 +23,6 @@ import org.junit.runner.RunWith import java.util.* -@RunWith(AndroidJUnit4::class) class UtilsTest { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/FinderTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/FinderTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/FinderTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/FinderTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/FinderTest.kt (date 1700607248374) @@ -36,7 +36,6 @@ import timber.log.Timber import java.util.* -@RunWith(AndroidJUnit4::class) class FinderTest : JvmTest() { @Test @Config(qualifiers = "en") Index: AnkiDroid/src/test/java/com/ichi2/libanki/MathJaxClozeTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/MathJaxClozeTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/MathJaxClozeTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/MathJaxClozeTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/MathJaxClozeTest.kt (date 1700607248382) @@ -12,7 +12,6 @@ import org.junit.Test import org.junit.runner.RunWith -@RunWith(AndroidJUnit4::class) @KotlinCleanup("removeFormattingFromMathjax was imported to stop bug in Kotlin: java.lang.NoSuchFieldError: INSTANCE") @KotlinCleanup("add testing function returning c.models.byName(\"Cloze\")") class MathJaxClozeTest : JvmTest() { Index: AnkiDroid/src/test/java/com/ichi2/libanki/CardTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/CardTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/CardTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/CardTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/CardTest.kt (date 1700607248359) @@ -31,7 +31,7 @@ import java.util.* import kotlin.test.assertNotNull -@RunWith(AndroidJUnit4::class) +@RunWith(AndroidJUnit4::class) // nextDueTest: Short Date Format is different class CardTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/TagsTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/TagsTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/TagsTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/TagsTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/TagsTest.kt (date 1700607248407) @@ -22,7 +22,6 @@ import org.junit.Test import org.junit.runner.RunWith -@RunWith(AndroidJUnit4::class) class TagsTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/StorageRustTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/StorageRustTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/StorageRustTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/StorageRustTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/StorageRustTest.kt (date 1700607248403) @@ -24,7 +24,6 @@ import org.junit.runner.RunWith import org.robolectric.annotation.Config -@RunWith(AndroidJUnit4::class) class StorageRustTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/ConfigTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/ConfigTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/ConfigTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/ConfigTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/ConfigTest.kt (date 1700607248367) @@ -26,8 +26,6 @@ import org.junit.Test import org.junit.jupiter.api.assertThrows import org.junit.runner.RunWith - -@RunWith(AndroidJUnit4::class) class ConfigTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/CollectionTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/CollectionTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/CollectionTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/CollectionTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/CollectionTest.kt (date 1700607248363) @@ -26,7 +26,7 @@ import org.junit.runner.RunWith import java.util.* -@RunWith(AndroidJUnit4::class) +@RunWith(AndroidJUnit4::class) // test_timestamps: AnkiDroidApp.instance class CollectionTest : JvmTest() { @Test fun editClozeGenerateCardsInSameDeck() { Index: AnkiDroid/src/test/java/com/ichi2/libanki/NoteWithColTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/NoteWithColTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/NoteWithColTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/NoteWithColTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/NoteWithColTest.kt (date 1700607248395) @@ -24,7 +24,6 @@ import org.junit.runner.RunWith import org.robolectric.annotation.Config -@RunWith(AndroidJUnit4::class) class NoteWithColTest : JvmTest() { @Test Index: AnkiDroid/src/test/java/com/ichi2/libanki/MetaTest.kt IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/AnkiDroid/src/test/java/com/ichi2/libanki/MetaTest.kt b/AnkiDroid/src/test/java/com/ichi2/libanki/MetaTest.kt --- a/AnkiDroid/src/test/java/com/ichi2/libanki/MetaTest.kt (revision 4ac94b14feb30004eb3d7f68cdb88f075b0fc459) +++ b/AnkiDroid/src/test/java/com/ichi2/libanki/MetaTest.kt (date 1700607248387) @@ -22,7 +22,6 @@ import org.junit.Test import org.junit.runner.RunWith -@RunWith(AndroidJUnit4::class) class MetaTest : JvmTest() { @Test fun ensureDatabaseIsInMemory() { ```

Result

Flaky tests sometimes failing with

'byte[][] net.ankiweb.rsdroid.NativeMethods.openBackend(byte[])'
java.lang.UnsatisfiedLinkError: 'byte[][] net.ankiweb.rsdroid.NativeMethods.openBackend(byte[])'
    at net.ankiweb.rsdroid.NativeMethods.openBackend(Native Method)
    at net.ankiweb.rsdroid.Backend.<init>(Backend.kt:81)

This is not a failure to call RustBackendLoader.ensureSetup(), if ensureSetup() is forced to be called twice, it fails Native Library /private/var/folders/ym/nqynp93d4j74dpzw20sq4wq00000gn/T/librsdroid-b7f0b01dd98e063d7f3e09f8a0a0979aafe9dc0c.dylib already loaded in another classloader

Classloaders

There are two classloaders in play:

Inside a JvmTest, if you use the non-default (SandboxClassLoader) to load RustBackendLoader:

Inside an AndroidJUnitTest: if you use the non-default AppClassLoader:

Debug info

HEAD is 4ac94b14feb30004eb3d7f68cdb88f075b0fc459

Research
david-allison commented 10 months ago

One likely workaround would be to split the following into different modules:

dae commented 10 months ago

I hit something similar to this in the past. From maybeSetupBackend():

            // We must make sure not to load the backend library into a test running outside
            // the Robolectric classloader, or subsequent Robolectric tests that run in this
            // process will be unable to make calls into the backend.

Changing forkEvery = 40 to forkEvery = 1 seems to fix it, but impacts performance. https://docs.gradle.org/current/userguide/jvm_test_suite_plugin.html is probably the path forward

david-allison commented 10 months ago

https://docs.gradle.org/current/userguide/jvm_test_suite_plugin.html is probably the path forward

Tried adding: ``` testing { suites { integrationTest(JvmTestSuite) { dependencies { implementation project() } targets { all { testTask.configure { shouldRunAfter(JvmTestSuite) } } } } } } ```

This creates a non-Android sourceset: integrationTest, which is ignored by Android Studio

Alternatives

dae commented 10 months ago

This is what ChatGPT suggested - I have not tried it myself:

plugins {
    id 'java'
}

testing {
    suites {
        robolectric(JvmTestSuite) {
            // Configure robolectric test suite specifics
        }
        javaUnit(JvmTestSuite) {
            // Configure java unit test suite specifics
        }
    }
}

sourceSets {
    main {
        java {
            srcDir 'src/main/java'
        }
    }
    robolectric {
        java {
            srcDir 'src/test/robolectric'
        }
    }
    javaUnit {
        java {
            srcDir 'src/test/javaunit'
        }
    }
}

testing.suites.robolectric {
    useJUnitJupiter()
    dependencies {
        implementation project.sourceSets.main.output
        implementation sourceSets.robolectric.output
        // Add your Robolectric-specific dependencies here
    }
}

testing.suites.javaUnit {
    useJUnitJupiter()
    dependencies {
        implementation project.sourceSets.main.output
        implementation sourceSets.javaUnit.output
        // Add your Java unit test-specific dependencies here
    }
}

Apparently after that, you would be able to do ./gradlew javaUnitTest, or robolectricTest, or just test to do them both.

mikehardy commented 10 months ago

As much as I'm a devotee of LudditeGPT the suggestion earlier to split tests, and the suggestion from our ML overlords seems like the right way to work around a different root classloader issue without getting so clever that we can't maintain it

david-allison commented 10 months ago

Non-Android source sets detected in ":Anki-Android":

Gradle source sets ignored: robolectric, javaUnit, main.

Unverified research: test/androidTest seem fairly heavily hardcoded: https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/VariantManager.kt;l=690-736?q=setUpSourceSet&ss=android-studio%2Fplatform%2Ftools%2Fbase

dae commented 10 months ago

Ok, I think I get you - it's an IDE issue you're trying to work around, not a gradle one? Does AndroidStudio cope fine with a pure-Java module? If not, maybe JUnit4's categories could be used so that an upgrade to JUnit5 is not required?

github-actions[bot] commented 7 months ago

Hello 👋, this issue has been opened for more than 3 months with no activity on it. If the issue is still here, please keep in mind that we need community support and help to fix it! Just comment something like still searching for solutions and if you found one, please open a pull request! You have 7 days until this gets closed automatically