search

chaquo / chaquopy

Chaquopy: the Python SDK for Android
https://chaquo.com/chaquopy/
MIT License
783 stars 130 forks source link

Numba error: resolveAArch64Relocation: assertion "static_cast<int64_t>(Result) >= INT32_MIN && static_cast<int64_t>(Result) <= UINT32_MAX" failed #527

Open chhsiao121 opened 3 years ago

chhsiao121 commented 3 years ago

Chaquopy version

classpath "com.chaquo.python:gradle:9.1.0"

Devices or emulators where the issue happens

HTC Mobile Phones and samsung tablet

Relevant parts of your code

Android code:

private float[] callPythonCode(String path) {
        Python py = Python.getInstance();
        PyObject obj1 = py.getModule("mfcc").callAttr("mfcc", path);
        JavaBean javaBean = obj1.toJava(JavaBean.class);
        return javaBean.getData();
    }
}

Python code

<mfcc.py>
from java import jclass, jarray, jfloat
import librosa
import numpy as np
import _multiprocessing
_multiprocessing.sem_unlink = None

def mfcc(wavfile):
    y, _ = librosa.load(wavfile)
    S = np.abs(librosa.stft(y, n_fft=512))
    p = librosa.amplitude_to_db(S, ref=np.max)
    print("mfcc working")
    tmp = np.zeros([256, 128])
    print('Python: image shape = 256, ',p.shape[1])
    if p.shape[1] > 128:
        tmp[:256, :128] = p[:256, :128]
    else:
        tmp[:256, :p.shape[1]] = p[:256, :p.shape[1]]
    tmp = (tmp+40)
    tmp = tmp/40.0
    tmp = tmp.flatten()
    tmp = np.float32(tmp)
    print("JB working")
    JavaBean = jclass("com.cgh.org.audio.TFlite.JavaBean")
    jb = JavaBean()
    jb.setData(jarray(jfloat)(tmp))
    return jb

Describe your issue

I want to call a function mfcc whitch is in my python code "mfcc.py" but when I use PyObject obj1 = py.getModule("mfcc").callAttr("mfcc", path); the app in my phone will quit unexpectedly. I don't know how to fix it.

error message:

 W/python.stderr: /data/user/0/com.ntust.cgh.tflite/files/chaquopy/AssetFinder/requirements/joblib/_multiprocessing_helpers.py:45: UserWarning: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770..  joblib will operate in serial mode
W/python.stderr: /data/user/0/com.ntust.cgh.tflite/files/chaquopy/AssetFinder/requirements/numba/npyufunc/parallel.py:300: UserWarning: Could not obtain multiprocessing lock due to OS level error: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
    A likely cause of this problem is '/dev/shm' is missing orread-only such that necessary semaphores cannot be written.
    *** The responsibility of ensuring multiprocessing safe access to this initialization sequence/module import is deferred to the user! ***

V/FA: Inactivity, disconnecting from the service
A/libc: /home/smith/git/chaquo/python/server/pypi/packages/chaquopy-llvm/build/8.0.0/py3-none-android_21_arm64_v8a/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:402: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry &, uint64_t, uint64_t, uint32_t, int64_t): assertion "static_cast<int64_t>(Result) >= INT32_MIN && static_cast<int64_t>(Result) <= UINT32_MAX" failed
    Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 20155 (Thread-6), pid 20088 (tust.cgh.tflite)
mhsmith commented 3 years ago

Does the problem also happen on the emulator?

Can you share an example of a WAV file which reproduces the problem? And does this happen with all WAV files, or only some?

Can you add some print statements before and after each of the librosa calls to narrow down which line is causing the crash?

mhsmith commented 3 years ago

If this is still a problem, please provide the requested information and I'll reopen the issue.

DellyMore commented 1 year ago

same issue on samsung a12 when trying to import (Only import) librosa (latest version at current date and resampy==0.3.1) while calling python instance function getModule. Python ver 3.10.6

Also problem not occured on Pixed 5 emulator device.

version: classpath 'com.chaquo.python:gradle:12.0.1'

Code of py file

import librosa // Crashes here
import numpy as np

def cutAndResampleAudio(oldSampleRate, newSampleRate, cutLenSec, rawData):
    newSize = int(cutLenSec * oldSampleRate)
    return librosa.resample(y=np.array(rawData[:newSize]), orig_sr=oldSampleRate, target_sr=newSampleRate)

def cutAudio(cutLenSec, rawData, sampleRate):
    return rawData[:int(cutLenSec * sampleRate)]

Kotlin code

private fun cutAndResampleAudioData(rawData: FloatArray, oldSampleRate: Int, newSampleRate: Int, newLenInSec: Float):FloatArray{
        val python = Python.getInstance()
        val pythonFile = python.getModule("downsampler") // Crashes here
        return pythonFile.callAttr("cutAndResampleAudio", oldSampleRate, newSampleRate, newLenInSec, rawData).toJava(FloatArray::class.java)
}
DellyMore commented 1 year ago

If this is still a problem, please provide the requested information and I'll reopen the issue.

will my info is enough?

mhsmith commented 1 year ago

@DellyMore: Thanks for the information, but I still can't reproduce the problem. With Chaquopy 12.0.1, and the following pip section in my build.gradle file:

                install "librosa==0.9.2"
                install "resampy==0.3.1"

... I can import librosa on both a Nexus 5X and a Pixel 7, with no problems apart from the expected warning about "platform lacks a functioning sem_open implementation".

Can you try your app on some other devices, and let me know which ones show this problem and which ones don't?

mhsmith commented 1 year ago

Originally posted by @technophile-exe in https://github.com/chaquo/chaquopy/issues/760#issue-1494486078

Chaquopy version

Relevant parts of your code

Python script

from scipy.io import wavfile
import noisereduce as nr

def main(input_path,output_path):
    print(str(input_path))
    rate, data = wavfile.read(input_path)
    # print('read file')
    # print(str(data.item(0)))
    reduced_noise = nr.reduce_noise(y=data, sr=rate)

    return "SUCCESS"

Java code

PyObject pyObject = py.getModule("myscript");
// Get filename returns path for the file on which we want to reduce noise. Ex: " /storage/emulated/0/Download/Ene_record.wav"
PyObject obj = pyObject.callAttr("main",getFilename(""),getFilename("nr"));

Describe your issue

I'm trying to use the python noisereduce library to reduce noise from wav file and write it back to storage. It throws the following error assertion "static_cast<int64_t>(Result) >= INT32_MIN && static_cast<int64_t>(Result) <= UINT32_MAX" failed A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 31043 (tsoundanalytics), pid 31043 (tsoundanalytics)

Logcat output

13661-13661 python.stderr           com.btp.heartsoundanalytics          W  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/joblib/_multiprocessing_helpers.py:46: UserWarning: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770..  joblib will operate in serial mode
13661-13661 python.stderr           com.btp.heartsoundanalytics          W  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/numba/npyufunc/parallel.py:300: UserWarning: Could not obtain multiprocessing lock due to OS level error: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
13661-13661 python.stderr           com.btp.heartsoundanalytics          W  A likely cause of this problem is '/dev/shm' is missing orread-only such that necessary semaphores cannot be written.
13661-13661 python.stderr           com.btp.heartsoundanalytics          W  *** The responsibility of ensuring multiprocessing safe access to this initialization sequence/module import is deferred to the user! ***
13661-13661 python.stderr           com.btp.heartsoundanalytics          W   
13661-13661 tsoundanalytics         com.btp.heartsoundanalytics          W  type=1400 audit(0.0:342572): avc: granted { execute } for path="/data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/stdlib-arm64-v8a/unicodedata.so" dev="dm-3" ino=5114037 scontext=u:r:untrusted_app:s0:c124,c259,c512,c768 tcontext=u:object_r:app_data_file:s0:c124,c259,c512,c768 tclass=file app=com.btp.heartsoundanalytics
13661-13661 tsoundanalytics         com.btp.heartsoundanalytics          W  type=1400 audit(0.0:342574): avc: granted { execute } for path="/data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/scipy/optimize/minpack2.so" dev="dm-3" ino=5114586 scontext=u:r:untrusted_app:s0:c124,c259,c512,c768 tcontext=u:object_r:app_data_file:s0:c124,c259,c512,c768 tclass=file app=com.btp.heartsoundanalytics
13661-13661 tsoundanalytics         com.btp.heartsoundanalytics          W  type=1400 audit(0.0:342576): avc: granted { execute } for path="/data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/scipy/linalg/_flapack.so" dev="dm-3" ino=5114596 scontext=u:r:untrusted_app:s0:c124,c259,c512,c768 tcontext=u:object_r:app_data_file:s0:c124,c259,c512,c768 tclass=file app=com.btp.heartsoundanalytics
13661-13661 tsoundanalytics         com.btp.heartsoundanalytics          W  type=1400 audit(0.0:342636): avc: granted { execute } for path="/data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/numba/npyufunc/workqueue.so" dev="dm-3" ino=5114955 scontext=u:r:untrusted_app:s0:c124,c259,c512,c768 tcontext=u:object_r:app_data_file:s0:c124,c259,c512,c768 tclass=file app=com.btp.heartsoundanalytics
--------- beginning of crash
13661-13661 libc                    com.btp.heartsoundanalytics          A  /home/smith/git/chaquo/python/server/pypi/packages/chaquopy-llvm/build/8.0.0/py3-none-android_21_arm64_v8a/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:402: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry &, uint64_t, uint64_t, uint32_t, int64_t): assertion "static_cast<int64_t>(Result) >= INT32_MIN && static_cast<int64_t>(Result) <= UINT32_MAX" failed
13661-13661 libc                    com.btp.heartsoundanalytics          A  Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 13661 (tsoundanalytics), pid 13661 (tsoundanalytics)
22353-22353 DEBUG                   pid-22353                            A  pid: 13661, tid: 13661, name: tsoundanalytics  >>> com.btp.heartsoundanalytics <<<
22353-22353 DEBUG                   pid-22353                            A        #02 pc 00000000019d1794  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/chaquopy/lib/libLLVM-8.so
22353-22353 DEBUG                   pid-22353                            A        #03 pc 00000000019bb7b0  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/chaquopy/lib/libLLVM-8.so
22353-22353 DEBUG                   pid-22353                            A        #04 pc 00000000019ba9f0  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/chaquopy/lib/libLLVM-8.so
22353-22353 DEBUG                   pid-22353                            A        #05 pc 000000000195bcec  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/chaquopy/lib/libLLVM-8.so
22353-22353 DEBUG                   pid-22353                            A        #06 pc 000000000002cf04  /data/data/com.btp.heartsoundanalytics/files/chaquopy/AssetFinder/requirements/llvmlite/binding/libllvmlite.so
22353-22353 DEBUG                   pid-22353                            A        #07 pc 000000000001ed54  /data/data/com.btp.heartsoundanalytics/files/chaquopy/bootstrap-native/arm64-v8a/_ctypes.so
---------------------------- PROCESS ENDED (13661) for package com.btp.heartsoundanalytics ----------------------------
 1569-2209  InputDispatcher         pid-1569                             E  channel '90d2b17 com.btp.heartsoundanalytics/com.btp.heartsoundanalytics.MainActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
mhsmith commented 1 year ago

Originally posted by @technophile-exe in https://github.com/chaquo/chaquopy/issues/760#issuecomment-1355324992

Hey @mhsmith I modified the code a bit and am now running it on a separate thread (previously I was running it on the main thread). I tried it on an emulator and I did not face the above issue but the issue persists for my device although it seems to be appearing randomly and many times it gets executed with any problem.

Modified code

private void reduceNoise(){
        try {
            String input_path = getFilename("");
            Log.d("File path",input_path);
            PyObject wavFile = py.getModule("scipy.io.wavfile");
            PyObject noiseReduce = py.getModule("noisereduce");
            Set<String> st = wavFile.keySet();
            List<PyObject> res = wavFile.callAttr("read",input_path).asList();
            PyObject  rate = res.get(0);
            PyObject data = res.get(1);
            Log.d("Wav read","Read data");
            PyObject reducedNoise = noiseReduce.callAttr("reduce_noise",new Kwarg("y",data),new Kwarg("sr",rate));
            PyObject out_path = PyObject.fromJava(getFilename("reduced"));
            wavFile.callAttr("write",out_path,rate,reducedNoise);
            Log.d("write complete","reduced noise file saved");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
mhsmith commented 1 year ago

@technophile-exe: If I simply list noisereduce in the pip section of my build.gradle file, I get the following warning during the build:

resampy 0.4.2 has requirement numba>=0.53, but you'll have numba 0.48.0 which is incompatible.

And the following crash during import:

10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/noisereduce/__init__.py", line 1, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/noisereduce/noisereduce.py", line 3, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/librosa/__init__.py", line 209, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/librosa/core/__init__.py", line 6, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/librosa/core/audio.py", line 12, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/resampy/__init__.py", line 7, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/resampy/core.py", line 11, in <module>
10:55:59.863  W    File "import.pxi", line 26, in java.chaquopy.import_override
10:55:59.863  W    File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/resampy/interpn.py", line 73, in <module>
10:55:59.863  W  TypeError: guvectorize() missing 1 required positional argument: 'signature'

So I assume you must have added some additional requirements, such as resampy==0.3.1 above? Please let me know exactly what you did.

mhsmith commented 1 year ago

I created pypi/packages/noisereduce/test/__init__.py with the following content:

from os.path import dirname
import unittest

class TestNoisereduce(unittest.TestCase):

    def test_basic(self):
        from scipy.io import wavfile
        import noisereduce as nr

        rate, data = wavfile.read(f"{dirname(__file__)}/fish.wav")
        reduced_noise = nr.reduce_noise(y=data, sr=rate)

And added fish.wav from here to the same directory.

Then, using the pkgtest app with Chaquopy 13.0.0, and the additional requirement resampy==0.3.1, I was able to run the test on both a Nexus 5X and a Pixel 7, with no problems apart from the expected warning about "platform lacks a functioning sem_open implementation".

@technophile-exe: Can you try your app on some other devices, and let me know which ones show this problem and which ones don't?

DellyMore commented 1 year ago

@DellyMore: Thanks for the information, but I still can't reproduce the problem. With Chaquopy 12.0.1, and the following pip section in my build.gradle file:

                install "librosa==0.9.2"
                install "resampy==0.3.1"

... I can import librosa on both a Nexus 5X and a Pixel 7, with no problems apart from the expected warning about "platform lacks a functioning sem_open implementation".

Can you try your app on some other devices, and let me know which ones show this problem and which ones don't?

Yes. I have 4 physical devices:

  1. Samsung A12
  2. Samsung A21
  3. Xiaomi mi 9 Lite
  4. Pixel 5. And 2 emulator devices:
    1. Pixel 5
    2. Pixel 3a Summary: In my case problem occurs only on Samsung devices, others works fine

Also i will paste my full defaultconfig for app

defaultConfig {
        minSdk 23
        targetSdk 33
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        consumerProguardFiles "consumer-rules.pro"
        sourceSets {
            main {
                python {
                    srcDirs = ["src/main/python"]
                }
            }
        }
        python {
            buildPython "/opt/homebrew/Caskroom/miniforge/base/envs/mlp/bin/python"
            buildPython "python3"
            pip {
                install "numpy"
                install "scipy"
                install "pandas"
                install "resampy==0.3.1"
                install "librosa"
            }
        }
        ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }
    }
mhsmith commented 1 year ago

Originally posted by @HughHMLee in https://github.com/chaquo/chaquopy/issues/21#issuecomment-1634264124

Hi, I'm trying to use librosa in my Android application targeting Android 11 (API 30). The app is able to run smoothly on Android 12 (API 31) and later, but crashes on Android 11 (API 30) and before.

The crash occurs when importing librosa:

/home/smith/git/chaquo/python/server/pypi/packages/chaquopy-llvm/build/8.0.0/py3-none-android_21_arm64_v8a/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:402: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry &, uint64_t, uint64_t, uint32_t, int64_t): assertion "static_cast<int64_t>(Result) >= INT32_MIN && static_cast<int64_t>(Result) <= UINT32_MAX" failed
Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3491 (e.myapplication), pid 3491 (e.myapplication)

I am warned that multiprocessing does not work, regardless of whether or not the app crashes.

The app does not crash when independently importing numba or llvmlite. If the app does not initially crash when importing librosa, it is able to run librosa functions smoothly.

Details of the environment:

Scenarios:

These scenarios occur both on the incomplete project I was given, as well as a new bare-bone project I created to test the issue.

Help and support would be greatly appreciated. I have attached relevant files from the bare-bone project below (note that they are largely unmodified stock files).

build.gradle:

plugins {
    id 'com.android.application'
    id 'com.chaquo.python'
}

android {
    namespace 'com.example.myapplication'
    compileSdk 33

    defaultConfig {
        applicationId "com.example.myapplication"
        minSdk 29
        targetSdk 33
        versionCode 1
        versionName "1.0"
        ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }

        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

        python {
            pip {
                install "decorator==5.1.1"
                install "resampy==0.3.1"
                install "librosa==0.9.2"
            }
        }
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}

dependencies {

    implementation 'androidx.appcompat:appcompat:1.6.1'
    implementation 'com.google.android.material:material:1.9.0'
    implementation 'androidx.constraintlayout:constraintlayout:2.1.4'
    testImplementation 'junit:junit:4.13.2'
    androidTestImplementation 'androidx.test.ext:junit:1.1.5'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.1'

    implementation 'com.chaquo.python.runtime:chaquopy:14.0.2'
}

MainActivity.java:

package com.example.myapplication;

import androidx.appcompat.app.AppCompatActivity;
import android.os.Bundle;

import com.chaquo.python.PyObject;
import com.chaquo.python.Python;
import com.chaquo.python.android.AndroidPlatform;

public class MainActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        Python.start(new AndroidPlatform(this));
        Python python = Python.getInstance();
        PyObject pyObject = python.getModule("foo");
    }
}

foo.py:

import librosa
mhsmith commented 1 year ago

@HughHMLee: Thanks for the information, and sorry for the slow reply.

Are your Windows and macOS machines x86-64 or ARM64?

Do your emulator system images match the architecture of the machine they're running on?

If you haven't already, please try reproducing the error on an x86-64 system image, and see if the error is exactly the same.

realzzq commented 3 months ago

i find out it work only in armv7 and python 3.8(python 2.7 must be invalid), right?

mhsmith commented 3 months ago

As you can see from the repository, we currently only have Numba builds for Python 3.8, but we do support all 4 Android ABIs.

If you're having problems, are you getting the same error message as in the title of this issue? If so, please answer the questions above. If not, please create a separate issue with full details.

[EDIT: I've already created a separate issue for the comment you posted elsewhere: #1154]