Sklearn import failed: multiprocessing.synchronize now imports successfully

[Flyfish233]

This exception occurs when updating the Chaquopy plugin version from 14.0.2 to 15.0.1 with no changes to any part of the code.

Chaquopy version

id("com.chaquo.python") version "15.0.1" apply false

Downgrading to id("com.chaquo.python") version "14.0.2" apply false will fix this problem. But I don't hope so.

Devices or emulators where the issue happens

Android 13, Wear OS Emulator

Relevant parts of your code

Gradle: install("scikit-learn == 1.1.3")

code:

val py = Python.getInstance()
val script = py.getModule("neuro")
val outputConsole = script.callAttr("ppg").toString()

It happens during the import, so the following minimal code snippet will throw an error.

neuro.py

import sklearn

def ppg():
    return "Okay"

Describe your issue

com.chaquo.python.PyException: Exception: multiprocessing.synchronize now imports successfully: check if its workaround can be removed
    at <python>.java.android.initialize_multiprocessing(__init__.py:163)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.joblib.disk.<module>(disk.py:17)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.joblib.logger.<module>(logger.py:20)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.joblib.func_inspect.<module>(func_inspect.py:18)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.joblib.memory.<module>(memory.py:31)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.joblib.<module>(__init__.py:114)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.sklearn.utils._joblib.<module>(_joblib.py:7)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.sklearn.utils.<module>(__init__.py:26)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.sklearn.base.<module>(base.py:17)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.sklearn.<module>(__init__.py:90)
    at <python>.java.chaquopy.import_override(import.pxi:26)
    at <python>.neuro.<module>(neuro.py:1)
    at <python>.importlib._bootstrap._call_with_frames_removed(<frozen importlib._bootstrap>:228)
    at <python>.importlib._bootstrap_external.exec_module(<frozen importlib._bootstrap_external>:850)
    at <python>.java.android.importer.exec_module(importer.py:634)
    at <python>.java.android.importer.exec_module(importer.py:721)
    at <python>.importlib._bootstrap._load_unlocked(<frozen importlib._bootstrap>:680)
    at <python>.importlib._bootstrap._find_and_load_unlocked(<frozen importlib._bootstrap>:986)
    at <python>.importlib._bootstrap._find_and_load(<frozen importlib._bootstrap>:1007)
    at <python>.importlib._bootstrap._gcd_import(<frozen importlib._bootstrap>:1030)
    at <python>.importlib.import_module(__init__.py:127)
    at <python>.chaquopy_java.Java_com_chaquo_python_Python_getModuleNative(chaquopy_java.pyx:129)
    at com.chaquo.python.Python.getModuleNative(Native Method)
    at com.chaquo.python.Python.getModule(Python.java:84)

[mhsmith]

You've closed the issue, does that mean you've solved the problem? If so, how?

[Flyfish233]

Yeah, scikit-learn == 1.1.3 won't work on latest Chaquopy. Upgrade to latest 1.3.2 on pypi solved this issue but 1.1.3 still not working.

[mhsmith]

The only Python version that has both of those scikit-learn versions is 3.9. Was that the Python version of your app, or did you change that at the same time?

[Flyfish233]

Yes, I have never used any version other than Python 3.9, locked Python version to 3.9.

[mhsmith]

This was caused by a change in Chaquopy 15 which applies the multiprocessing monkey patches when multiprocessing is first imported, rather than doing it during Python startup as before. But an older version of one of the monkey patches is built into our wheels of scikit-learn 1.1.3, and when that version is applied first, which will now happen if sklearn is imported before multiprocessing, Chaquopy's own version will fail.

However, we never actually needed the patch in that version of scikit-learn, because it was released through the pypi-13.1 repository and was therefore only visible to versions of Chaquopy which already had the patch. So I'll fix this by making new builds of version 1.1.3 with the patch removed.

Meanwhile, you can work around this by importing multiprocessing before importing sklearn.

[mhsmith]

I've now released new builds of scikit-learn 1.1.3, and saved the recipe in #1078. Thanks for the report.