search

k2-fsa / sherpa-onnx

Speech-to-text, text-to-speech, speaker recognition, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift, Dart, JavaScript, Flutter, Object Pascal, Lazarus, Rust
https://k2-fsa.github.io/sherpa/onnx/index.html
Apache License 2.0
3.35k stars 391 forks source link

Flutter (Dart) apps fail Apple's submission validation, likely due to libsherpa dylib being present in root of Frameworks folder #1172

Open BrutalCoding opened 2 months ago

BrutalCoding commented 2 months ago

Heya,

First off, many thanks for working on bringing this package to Dart. I have been working with this package since several months ago in Flutter in my own way and today I tried to check out your official Dart version on pub.dev - it's great!

TTS, STT, VAD (the new v5) are all working perfect on my physical iPhone 12 Pro, so the features are not an issue here.

There is an issue though when deploying an app for App Store because sherpa-onnx bundles the dynamic library libsherpa-onnx-c-api.dylib in the root of the Frameworks directory and that gets automatically rejected by Apple when validating an app app for submission. It must be packaged as a framework. Note: This is an issue on at least iOS, as I haven't tried sherpa-onnx (Dart plugin) on macOS yet, but it's likely an issue there too.

Misleading error message: ITMS-90426: Invalid Swift Support - The SwiftSupport folder is missing. Rebuild your app using the current public (GM) version of Xcode and resubmit it. (Doesn't help - already on latest version. See below for suspected real cause of error in this case).

Here's what I've tried:

Failed submission of Runner.ipa with Transporter.app

  1. Build app for release with flutter build ios
  2. Drag the newly built .ipa file into Apple's Transporter.app app During the validation stage, Transporter will show this error: image

Failed submission of Runner.xcarchive via XCode

  1. Find the Runner.xcarchive in Finder
  2. Double click to open it in Xcode (if I recall correctly). Alternatively, open the ios project in Xcode and click on "View" (top menu-bar) -> Organizer -> Drag and Drop the Runner.xcarchive file in there.
  3. Click on the archive in the Organizer pane and either click on "Validate App" or "Distribute App" (I tried both of course).
  4. Click through and when asked, tell Xcode you're uploading the app for App Store Connect (do not choose for Ad hoc, Enterprise or Debugging)
  5. All validity checks passed with success
  6. Quickly visit App Store Connect, visit your app there and check the TestFlight page. You'll get the impression that your app is almost done processing and ready to test because the new version is appearing there for a minute or so (it silently disappears).
  7. Check your e-mail inbox for Apple's e-mail, it's the same error: image

Identified issues

The error message is misleading, since I am running the latest Xcode and I have submitted it in all possible ways. This led me to investigate the contents of the Runner.xcarchive to see how sherpa-onnx is shipped with the app, and I found an issue here: image

We can not ship an app with a dylib thrown in the root of the Frameworks folder, instead, it must be packaged in it's own contained "framework" (just a folder structure with a couple files for metadata e.g. info.plist).

I'm sorry to not have the specific details at hand, I just wanted to highlight this issue before I'm fixing this. A quick work-around that I will try first is not fixing any source code, but by modifying the contents of the archive and re-attempting to upload the app for submission.

Better way would be to fix this in the source code of course. Some changes are likely required where you build with CMake against the C API, after producing the the shared library (the .dylib) we could add a few more lines to produce a xcframework folder and move the .dylib in there.

If there's anything unclear, ask away.

edit:

BrutalCoding commented 2 months ago

I will work on this issue and post an update tomorrow. I will also see if I can find and fix the issue in sherpa-onnx itself, if so, I'll open a PR of course.

csukuangfj commented 2 months ago

Thank you!

BrutalCoding commented 2 months ago

Quick update, I've written a script that fixes this issue. Script expects 1 argument that points to the file path of the affected .xcarchive and fixes it.

I haven't had time to look into the sherpa-onnx source itself, but I'm pretty confident that I can solve this here too.

If I can't find and fix the issue in sherpa-onnx itself in the upcoming days, I'll share my script. That way, we can fix it together in sherpa-onnx.

BrutalCoding commented 2 months ago

Here's the new .xcarchive content, showing that sherpa-onnx is now a good .framework instead of being absent (none listed). This is a screenshot of mine taken from one of the confirmation dialogs in Xcode > Organizer (Archives) > Distribute App (blue button):

image

BrutalCoding commented 2 months ago

Update: I discovered an issue regarding missing symbols during runtime when I tested my test app from TestFlight. I'll post here once I've figured it out.

Just adding this note for reference, it's related to the issue I am facing (the error message) and the observation I made (the .dylib in the Frameworks folder): https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-TROUBLESHOOTING_BUNDLE_ERRORS-EMBEDDED__DYLIB_FILES

image

Here are some more links with valuable info:

I'll need some more time to figure this out.

Okay, 1 hour after I placed this comment and it's fixed. What a pain to work with Apple's confusing dev requirements but here's what I did in short:

  1. Run the build-ios-shared.sh script
  2. Find and copy the libsherpa-onnx-c-api.dylib file.
  3. Navigate to sherpa-onnx/flutter/sherpa_onnx/sherpa_onnx_ios/ios/. Place the copied .dylib here.
  4. Still within this ios folder, create a SherpaOnnx.framework folder and add a proper Info.plist inside this new .framework folder.
  5. Now, use lipo and install_name_tool to convert libsherpa-onnx-c-api.dylib into SherpaOnnx (no file extension). Move this SherpaOnnx binary into the .framework folder.
  6. Update the podspec file by adding a vendored_frameworks that is referring to this new .framework. Also, remove the existing vendored_libraries line.
  7. Update the dart file to replace the dynamic library lookup for iOS from DynamicLibrary.open("libsherpa-onnx-c-api.dylib") to DynamicLibrary.process(), this method will be able to find all global symbols that are running along with the app.

I will review my own changes over the next days and will try to keep it to a mimimum.

helang1991 commented 2 months ago

Update: I discovered an issue regarding missing symbols during runtime when I tested my test app from TestFlight. I'll post here once I've figured it out.

Just adding this note for reference, it's related to the issue I am facing (the error message) and the observation I made (the .dylib in the Frameworks folder): https://developer.apple.com/library/archive/technotes/tn2435/_index.html#//apple_ref/doc/uid/DTS40017543-CH1-TROUBLESHOOTING_BUNDLE_ERRORS-EMBEDDED__DYLIB_FILES

image

Here are some more links with valuable info:

I'll need some more time to figure this out.

Okay, 1 hour after I placed this comment and it's fixed. What a pain to work with Apple's confusing dev requirements but here's what I did in short:

  1. Run the build-ios-shared.sh script
  2. Find and copy the libsherpa-onnx-c-api.dylib file.
  3. Navigate to sherpa-onnx/flutter/sherpa_onnx/sherpa_onnx_ios/ios/. Place the copied .dylib here.
  4. Still within this ios folder, create a SherpaOnnx.framework folder and add a proper Info.plist inside this new .framework folder.
  5. Now, use lipo and install_name_tool to convert libsherpa-onnx-c-api.dylib into SherpaOnnx (no file extension). Move this SherpaOnnx binary into the .framework folder.
  6. Update the podspec file by adding a vendored_frameworks that is referring to this new .framework. Also, remove the existing vendored_libraries line.
  7. Update the dart file to replace the dynamic library lookup for iOS from DynamicLibrary.open("libsherpa-onnx-c-api.dylib") to DynamicLibrary.process(), this method will be able to find all global symbols that are running along with the app.

I will review my own changes over the next days and will try to keep it to a mimimum.

Hello, I followed your method, but encountered a problem. Can you share detail steps, like screenshots

helang1991 commented 2 months ago

@BrutalCoding Hello, I found a problem that uploading to testFlight

image
eschmidbauer commented 2 months ago

any update on this? i'm trying to build for iOS and encountering same error. here are some steps i outlined based on the resources in this thread:

bash build-ios-shared.sh

cd build-ios-shared/ios-arm64-simulator/

lipo -create libsherpa-onnx-c-api.dylib -output SherpaOnnx