k2-fsa / sherpa-onnx

Speech-to-text, text-to-speech, speaker diarization, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift, Dart, JavaScript, Flutter, Object Pascal, Lazarus, Rust

https://k2-fsa.github.io/sherpa/onnx/index.html

Apache License 2.0

3.72k stars 431 forks source link

Flutter (Dart) apps fail Apple's submission validation, likely due to libsherpa dylib being present in root of Frameworks folder #1172

Closed BrutalCoding closed 2 weeks ago

BrutalCoding commented 4 months ago

Heya,

First off, many thanks for working on bringing this package to Dart. I have been working with this package since several months ago in Flutter in my own way and today I tried to check out your official Dart version on pub.dev - it's great!

TTS, STT, VAD (the new v5) are all working perfect on my physical iPhone 12 Pro, so the features are not an issue here.

There is an issue though when deploying an app for App Store because sherpa-onnx bundles the dynamic library libsherpa-onnx-c-api.dylib in the root of the Frameworks directory and that gets automatically rejected by Apple when validating an app app for submission. It must be packaged as a framework. Note: This is an issue on at least iOS, as I haven't tried sherpa-onnx (Dart plugin) on macOS yet, but it's likely an issue there too.

Misleading error message: ITMS-90426: Invalid Swift Support - The SwiftSupport folder is missing. Rebuild your app using the current public (GM) version of Xcode and resubmit it. (Doesn't help - already on latest version. See below for suspected real cause of error in this case).

Here's what I've tried:

Failed submission of Runner.ipa with Transporter.app

Build app for release with flutter build ios
Drag the newly built .ipa file into Apple's Transporter.app app During the validation stage, Transporter will show this error:

Failed submission of Runner.xcarchive via XCode

Find the Runner.xcarchive in Finder
Double click to open it in Xcode (if I recall correctly). Alternatively, open the ios project in Xcode and click on "View" (top menu-bar) -> Organizer -> Drag and Drop the Runner.xcarchive file in there.
Click on the archive in the Organizer pane and either click on "Validate App" or "Distribute App" (I tried both of course).
Click through and when asked, tell Xcode you're uploading the app for App Store Connect (do not choose for Ad hoc, Enterprise or Debugging)
All validity checks passed with success
Quickly visit App Store Connect, visit your app there and check the TestFlight page. You'll get the impression that your app is almost done processing and ready to test because the new version is appearing there for a minute or so (it silently disappears).
Check your e-mail inbox for Apple's e-mail, it's the same error:

Identified issues

The error message is misleading, since I am running the latest Xcode and I have submitted it in all possible ways. This led me to investigate the contents of the Runner.xcarchive to see how sherpa-onnx is shipped with the app, and I found an issue here:

We can not ship an app with a dylib thrown in the root of the Frameworks folder, instead, it must be packaged in it's own contained "framework" (just a folder structure with a couple files for metadata e.g. info.plist).

I'm sorry to not have the specific details at hand, I just wanted to highlight this issue before I'm fixing this. A quick work-around that I will try first is not fixing any source code, but by modifying the contents of the archive and re-attempting to upload the app for submission.

Better way would be to fix this in the source code of course. Some changes are likely required where you build with CMake against the C API, after producing the the shared library (the .dylib) we could add a few more lines to produce a xcframework folder and move the .dylib in there.

If there's anything unclear, ask away.

edit:

Fixed typo from '.xarchive' to '.xcarchive'
Replaced '' with 'Runner' for clarity

BrutalCoding commented 4 months ago

I will work on this issue and post an update tomorrow. I will also see if I can find and fix the issue in sherpa-onnx itself, if so, I'll open a PR of course.

csukuangfj commented 4 months ago

Thank you!

BrutalCoding commented 4 months ago

Quick update, I've written a script that fixes this issue. Script expects 1 argument that points to the file path of the affected .xcarchive and fixes it.

I haven't had time to look into the sherpa-onnx source itself, but I'm pretty confident that I can solve this here too.

If I can't find and fix the issue in sherpa-onnx itself in the upcoming days, I'll share my script. That way, we can fix it together in sherpa-onnx.

BrutalCoding commented 4 months ago

Here's the new .xcarchive content, showing that sherpa-onnx is now a good .framework instead of being absent (none listed). This is a screenshot of mine taken from one of the confirmation dialogs in Xcode > Organizer (Archives) > Distribute App (blue button):

BrutalCoding commented 4 months ago

Update: I discovered an issue regarding missing symbols during runtime when I tested my test app from TestFlight. I'll post here once I've figured it out.

Just adding this note for reference, it's related to the issue I am facing (the error message) and the observation I made (the .dylib in the Frameworks folder): https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-TROUBLESHOOTING_BUNDLE_ERRORS-EMBEDDED__DYLIB_FILES

Here are some more links with valuable info:

This issue is related to the same type of issue we're facing. Link: https://github.com/flutter/flutter/issues/140544#issuecomment-1874703095
I personally faced this issue earlier too during the development of my Flutter plugin, and solved it too. Yet, I can't fully grasp the issue right now. Link: https://github.com/BrutalCoding/aub.ai/blob/b16f7e5f9d317d83b38a5814afb1556c8de8a3d3/brutalcoding_scripts/autoupdate_ios.sh#L66C3-L66C7. I looked and changed things around files such as your sherpa_onnx_ios.podspec, sherpa_onnx.dart (e.g. trying DynamicLibrary.process() instead), and I also looked at your cmake stuff at build-ios-shared.sh/build-ios.sh etc.

I'll need some more time to figure this out.

Okay, 1 hour after I placed this comment and it's fixed. What a pain to work with Apple's confusing dev requirements but here's what I did in short:

Run the build-ios-shared.sh script
Find and copy the libsherpa-onnx-c-api.dylib file.
Navigate to sherpa-onnx/flutter/sherpa_onnx/sherpa_onnx_ios/ios/. Place the copied .dylib here.
Still within this ios folder, create a SherpaOnnx.framework folder and add a proper Info.plist inside this new .framework folder.
Now, use lipo and install_name_tool to convert libsherpa-onnx-c-api.dylib into SherpaOnnx (no file extension). Move this SherpaOnnx binary into the .framework folder.
Update the podspec file by adding a vendored_frameworks that is referring to this new .framework. Also, remove the existing vendored_libraries line.
Update the dart file to replace the dynamic library lookup for iOS from DynamicLibrary.open("libsherpa-onnx-c-api.dylib") to DynamicLibrary.process(), this method will be able to find all global symbols that are running along with the app.

I will review my own changes over the next days and will try to keep it to a mimimum.

helang1991 commented 3 months ago

Update: I discovered an issue regarding missing symbols during runtime when I tested my test app from TestFlight. I'll post here once I've figured it out.

Just adding this note for reference, it's related to the issue I am facing (the error message) and the observation I made (the .dylib in the Frameworks folder): https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-TROUBLESHOOTING_BUNDLE_ERRORS-EMBEDDED__DYLIB_FILES

Here are some more links with valuable info:

This issue is related to the same type of issue we're facing. Link: Native Assets should use Framework instead of dylib on iOS. flutter/flutter#140544 (comment)

I personally faced this issue earlier too during the development of my Flutter plugin, and solved it too. Yet, I can't fully grasp the issue right now. Link: https://github.com/BrutalCoding/aub.ai/blob/b16f7e5f9d317d83b38a5814afb1556c8de8a3d3/brutalcoding_scripts/autoupdate_ios.sh#L66C3-L66C7. I looked and changed things around files such as your sherpa_onnx_ios.podspec, sherpa_onnx.dart (e.g. trying DynamicLibrary.process() instead), and I also looked at your cmake stuff at build-ios-shared.sh/build-ios.sh etc.

I'll need some more time to figure this out.

Okay, 1 hour after I placed this comment and it's fixed. What a pain to work with Apple's confusing dev requirements but here's what I did in short:

Run the build-ios-shared.sh script

Find and copy the libsherpa-onnx-c-api.dylib file.

Navigate to sherpa-onnx/flutter/sherpa_onnx/sherpa_onnx_ios/ios/. Place the copied .dylib here.

Still within this ios folder, create a SherpaOnnx.framework folder and add a proper Info.plist inside this new .framework folder.

Now, use lipo and install_name_tool to convert libsherpa-onnx-c-api.dylib into SherpaOnnx (no file extension). Move this SherpaOnnx binary into the .framework folder.

Update the podspec file by adding a vendored_frameworks that is referring to this new .framework. Also, remove the existing vendored_libraries line.

Update the dart file to replace the dynamic library lookup for iOS from DynamicLibrary.open("libsherpa-onnx-c-api.dylib") to DynamicLibrary.process(), this method will be able to find all global symbols that are running along with the app.

I will review my own changes over the next days and will try to keep it to a mimimum.

Hello, I followed your method, but encountered a problem. Can you share detail steps, like screenshots

helang1991 commented 3 months ago

@BrutalCoding Hello, I found a problem that uploading to testFlight

eschmidbauer commented 3 months ago

any update on this? i'm trying to build for iOS and encountering same error. here are some steps i outlined based on the resources in this thread:

bash build-ios-shared.sh

cd build-ios-shared/ios-arm64-simulator/

lipo -create libsherpa-onnx-c-api.dylib -output SherpaOnnx

hantrungkien commented 1 month ago

@BrutalCoding I did like https://github.com/k2-fsa/sherpa-onnx/issues/1283#issuecomment-2381113506 but I got a error https://github.com/k2-fsa/sherpa-onnx/issues/1283#issuecomment-2437252099

Do you have any solution for that? Many thanks!

hantrungkien commented 1 month ago

This is the complete solution for everyone having the same problem as me:

Step 1.

git clone https://github.com/k2-fsa/sherpa-onnx.git && \
cd sherpa-onnx && \
sh build-ios-shared.sh && \
cd build-ios-shared

Step 2.

mkdir ios-arm64_x86_64-simulator && \
lipo -create \
ios-arm64-simulator/libsherpa-onnx-c-api.dylib \
ios-x86_64-simulator/libsherpa-onnx-c-api.dylib \
-output ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.dylib

Step 3.

cd ios-arm64 && \
mkdir libsherpa-onnx-c-api.framework && \
lipo -create libsherpa-onnx-c-api.dylib -output libsherpa-onnx-c-api && \
mv libsherpa-onnx-c-api libsherpa-onnx-c-api.framework/ && \
rm -rf libsherpa-onnx-c-api.dylib && \
cd libsherpa-onnx-c-api.framework && \
install_name_tool -change @rpath/libsherpa-onnx-c-api.dylib @rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api libsherpa-onnx-c-api && \
install_name_tool -id "@rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api" libsherpa-onnx-c-api && \
chmod +x libsherpa-onnx-c-api && \
cd .. && \
cd ..

Step 4.

cd ios-arm64_x86_64-simulator && \
mkdir libsherpa-onnx-c-api.framework && \
lipo -create libsherpa-onnx-c-api.dylib -output libsherpa-onnx-c-api && \
mv libsherpa-onnx-c-api libsherpa-onnx-c-api.framework/ && \
rm -rf libsherpa-onnx-c-api.dylib && \
cd libsherpa-onnx-c-api.framework && \
install_name_tool -change @rpath/libsherpa-onnx-c-api.dylib @rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api libsherpa-onnx-c-api && \
install_name_tool -id "@rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api" libsherpa-onnx-c-api && \
chmod +x libsherpa-onnx-c-api && \
cd .. && \
cd ..

Step 5. Add Info.plist

ios-arm64/libsherpa-onnx-c-api.framework/Info.plist
ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.framework/Info.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>BuildMachineOSBuild</key>
    <string>{YOUR_MACHINE_BUILD}</string>
    <key>CFBundleDevelopmentRegion</key>
    <string>en</string>
    <key>CFBundleExecutable</key>
    <string>libsherpa-onnx-c-api</string>
    <key>CFBundleIdentifier</key>
    <string>{YOUR_BUNDLE_ID}</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleName</key>
    <string>libsherpa-onnx-c-api</string>
    <key>CFBundlePackageType</key>
    <string>FMWK</string>
    <key>CFBundleShortVersionString</key>
    <string>1.0.0</string>
    <key>CFBundleSupportedPlatforms</key>
    <array>
        <string>iPhoneOS</string>
    </array>
    <key>CFBundleVersion</key>
    <string>1</string>
    <key>MinimumOSVersion</key>
    <string>{YOUR_MINIMUM_VERSION}</string>
</dict>
</plist>

Step 7.

xcodebuild -create-xcframework \
-framework ios-arm64/libsherpa-onnx-c-api.framework \
-framework ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.framework \
-output sherpa_onnx.xcframework

Step 8. Copy sherpa_onnx.xcframework to ios/Frameworks and add it to Runner

Step 9. Only get 2 packages from sherpa-onnx repository

sherpa_onnx
sherpa_onnx_android

Step 10. Modify

`sherpa_onnx`

pubspec.yaml
- Remove other dependencies except sherpa_onnx_android

dependencies:
  ffi: ^2.1.0
  flutter:
    sdk: flutter

  sherpa_onnx_android:
    path: sherpa_onnx_android

dev_dependencies:
  flutter_lints: ^3.0.0

flutter:
  plugin:
    platforms:
      android:
        default_package: sherpa_onnx_android

lib/sherpa_onnx.dart

void initBindings() {
  SherpaOnnxBindings.init(Platform.isAndroid
      ? DynamicLibrary.open('libsherpa-onnx-c-api.so')
      : DynamicLibrary.open(
          'libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api'));
}

`sherpa_onnx_android`

Move sherpa_onnx_android to inside sherpa_onnx

w-rui commented 3 weeks ago

@hantrungkien @csukuangfj 有计划什么时候推送线上更新吗？手动改这个挺麻烦的，更新也不方便

hantrungkien commented 3 weeks ago

@w-rui You can fork this repo and make your script follow as my guide to automatically build and publish your library.

csukuangfj commented 3 weeks ago

@hantrungkien @csukuangfj 有计划什么时候推送线上更新吗？手动改这个挺麻烦的，更新也不方便

没计划。

因为，pub.dev限制每个包解压后，不超过100MB。我们没法把simulator相关的库也上传到pub.dev

w-rui commented 3 weeks ago

明白了，想要更新线上，只能将 framework 相关的大文件上传 cocopods，然后才能上传 pub。不过你们可以考虑将上文的手动 script 流程写到 flutter/build_ios_framework.sh 之类的。多谢

csukuangfj commented 2 weeks ago

@hantrungkien @csukuangfj 有计划什么时候推送线上更新吗？手动改这个挺麻烦的，更新也不方便

不需要手动改了。目前最新版 v1.10.31 支持了. 不需要做任何代码改动。@w-rui

csukuangfj commented 2 weeks ago

This is the complete solution for everyone having the same problem as me:

Step 1.

git clone https://github.com/k2-fsa/sherpa-onnx.git && \
cd sherpa-onnx && \
sh build-ios-shared.sh && \
cd build-ios-shared

Step 2.

mkdir ios-arm64_x86_64-simulator && \
lipo -create \
ios-arm64-simulator/libsherpa-onnx-c-api.dylib \
ios-x86_64-simulator/libsherpa-onnx-c-api.dylib \
-output ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.dylib

Step 3.

cd ios-arm64 && \
mkdir libsherpa-onnx-c-api.framework && \
lipo -create libsherpa-onnx-c-api.dylib -output libsherpa-onnx-c-api && \
mv libsherpa-onnx-c-api libsherpa-onnx-c-api.framework/ && \
rm -rf libsherpa-onnx-c-api.dylib && \
cd libsherpa-onnx-c-api.framework && \
install_name_tool -change @rpath/libsherpa-onnx-c-api.dylib @rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api libsherpa-onnx-c-api && \
install_name_tool -id "@rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api" libsherpa-onnx-c-api && \
chmod +x libsherpa-onnx-c-api && \
cd .. && \
cd ..

Step 4.

cd ios-arm64_x86_64-simulator && \
mkdir libsherpa-onnx-c-api.framework && \
lipo -create libsherpa-onnx-c-api.dylib -output libsherpa-onnx-c-api && \
mv libsherpa-onnx-c-api libsherpa-onnx-c-api.framework/ && \
rm -rf libsherpa-onnx-c-api.dylib && \
cd libsherpa-onnx-c-api.framework && \
install_name_tool -change @rpath/libsherpa-onnx-c-api.dylib @rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api libsherpa-onnx-c-api && \
install_name_tool -id "@rpath/libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api" libsherpa-onnx-c-api && \
chmod +x libsherpa-onnx-c-api && \
cd .. && \
cd ..

Step 5. Add Info.plist

ios-arm64/libsherpa-onnx-c-api.framework/Info.plist
ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.framework/Info.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>BuildMachineOSBuild</key>
    <string>{YOUR_MACHINE_BUILD}</string>
    <key>CFBundleDevelopmentRegion</key>
    <string>en</string>
    <key>CFBundleExecutable</key>
    <string>libsherpa-onnx-c-api</string>
    <key>CFBundleIdentifier</key>
    <string>{YOUR_BUNDLE_ID}</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleName</key>
    <string>libsherpa-onnx-c-api</string>
    <key>CFBundlePackageType</key>
    <string>FMWK</string>
    <key>CFBundleShortVersionString</key>
    <string>1.0.0</string>
    <key>CFBundleSupportedPlatforms</key>
    <array>
        <string>iPhoneOS</string>
    </array>
    <key>CFBundleVersion</key>
    <string>1</string>
    <key>MinimumOSVersion</key>
    <string>{YOUR_MINIMUM_VERSION}</string>
</dict>
</plist>

Step 7.

xcodebuild -create-xcframework \
-framework ios-arm64/libsherpa-onnx-c-api.framework \
-framework ios-arm64_x86_64-simulator/libsherpa-onnx-c-api.framework \
-output sherpa_onnx.xcframework

Step 8. Copy sherpa_onnx.xcframework to ios/Frameworks and add it to Runner

Step 9. Only get 2 packages from sherpa-onnx repository

sherpa_onnx
sherpa_onnx_android

Step 10. Modify

`sherpa_onnx`

pubspec.yaml

Remove other dependencies except sherpa_onnx_android

dependencies:
  ffi: ^2.1.0
  flutter:
    sdk: flutter

  sherpa_onnx_android:
    path: sherpa_onnx_android

dev_dependencies:
  flutter_lints: ^3.0.0

flutter:
  plugin:
    platforms:
      android:
        default_package: sherpa_onnx_android

lib/sherpa_onnx.dart

void initBindings() {
  SherpaOnnxBindings.init(Platform.isAndroid
      ? DynamicLibrary.open('libsherpa-onnx-c-api.so')
      : DynamicLibrary.open(
          'libsherpa-onnx-c-api.framework/libsherpa-onnx-c-api'));
}

`sherpa_onnx_android`

Move sherpa_onnx_android to inside sherpa_onnx

@hantrungkien Thank you for sharing.

We have integrated it into our latest release v1.10.31

You don't need to change any code now.

hantrungkien commented 2 weeks ago

@csukuangfj Many thanks and what is the commit that's related with v1.10.31? I ask because I don't find it.

csukuangfj commented 2 weeks ago

@csukuangfj Many thanks and what is the commit that's related with v1.10.31? I ask because I don't find it.

please see #1547

csukuangfj commented 2 weeks ago

Does it work now?

hantrungkien commented 2 weeks ago

@csukuangfj sorry, I don't have time to upgrade to v1.10.31 yet. Maybe I will do that in next week...