search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.25k stars 1.58k forks source link

Does FFI support iOS static library？ #44328

Closed JackPanda8 closed 3 years ago

JackPanda8 commented 3 years ago

as the official doc : Symbols from a statically linked library can be loaded using DynamicLibrary.executable or DynamicLibrary.process.

but i got the error: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, pause_all_task): symbol not found)

-------------here is my code--------------------------------

import 'dart:ffi' as ffi;
import 'dart:io';
import 'package:ffi/ffi.dart';

// C function signatures
typedef pause_all_task_t = ffi.Void Function();
typedef get_cache_instance = ffi.Pointer<ffi.Void> Function();

// Dart function signatures
typedef PauseALLTask = void Function();
typedef GetCacheInstance = ffi.Pointer<ffi.Void> Function();

// Getting a library that holds needed symbols
ffi.DynamicLibrary _lib = Platform.isAndroid
        ? ffi.DynamicLibrary.open('libflash-downloader-lib.so')
        : ffi.DynamicLibrary.process();

// Looking for the functions
final PauseALLTask pauseAllTaskF = _lib
        .lookup<ffi.NativeFunction<pause_all_task_t>>('pause_all_task')
        .asFunction();
final GetCacheInstance getCacheInstanceF = _lib
        .lookup<ffi.NativeFunction<get_cache_instance>>('get_cache_instance')
        .asFunction();
#include "dart_ffi_adapter.h"
//#include "YKCacheCenter.h"
extern "C" {
__attribute__((visibility("default"))) __attribute__((used))
void* get_cache_instance(void) {
    printf("my_download:调用GetCacheInstance构造单例");
    void* p;
    return p;
}
__attribute__((visibility("default"))) __attribute__((used))
void pause_all_task(void) {
    //    yk_cache::IYKCache *cache = yk_cache::GetCacheInstance();
    //    cache->pauseAllTask();
    printf("my_download:调用pause_all_task暂停所有下载中的任务");
}
__attribute__((visibility("default"))) __attribute__((used))
void ff_pause_all_task(){
    printf("my_download:call ff_pause_all_task");
}
}
mkustermann commented 3 years ago

/cc @dcharkes

dcharkes commented 3 years ago

Hi @JackPanda8,

Does FFI support iOS static library？

Yes, following the steps from the documentation locally works for me.

Please take a look at https://github.com/flutter/flutter/issues/62666#issuecomment-668377567

And please double check that Xcode statically links the C code:

On iOS, you need to tell Xcode to statically link the file:

In Xcode, open Runner.xcworkspace. Add the C/C++/Objective-C/Swift source files to the Xcode project.

https://flutter.dev/docs/development/platform-integration/c-interop#step-2-add-cc-sources

If the problem still persists, please provide a minimal reproduction.

bitsydarel commented 3 years ago

Header and universal static library are added

Capture d’écran 2021-05-20 à 16 46 15

Strip style set to non-global

Capture d’écran 2021-05-20 à 16 47 29

Even added the attribute about visibility

Capture d’écran 2021-05-20 à 16 48 42

It's not working even debug build on simulator

Capture d’écran 2021-05-20 à 16 51 06

Even the iOS swift app delegate can see and run it without problem

Capture d’écran 2021-05-20 à 17 07 34

the universal static library support all the cpu

Capture d’écran 2021-05-20 à 17 09 18

even create a test c file

Capture d’écran 2021-05-20 à 17 29 58

added pseudo code that internally call the library function

Capture d’écran 2021-05-20 à 17 31 21 Capture d’écran 2021-05-20 à 17 31 44 Capture d’écran 2021-05-20 à 17 35 53

The number "3" is successfully printed in the console so the FPDF_DestroyLibrary symbol is definetly not stripped.

Capture d’écran 2021-05-20 à 17 35 15

But still dart can't see it...

Capture d’écran 2021-05-20 à 17 36 37
bitsydarel commented 3 years ago

Here are the necessary files need to to replicate https://drive.google.com/file/d/1tpI_StOh7ffPHjVapJJ34dMg2tJka45E/view?usp=sharing

bitsydarel commented 3 years ago

Any update or comment?

dcharkes commented 3 years ago

Hi @bitsydarel, I have not yet had time to look into this.

The last time I tried to do static linking for iOS and MacOS I ran into similar problems. I ended up using dynamic libraries instead.

Does using dynamic libraries work for you in the mean time? (Until I have time to dig into this.)

bitsydarel commented 3 years ago

Hi @dcharkes not really as those libraries are provided to us, and I have not seen usecase of creating fat library as dynamic libraries (for simulator and device).

bitsydarel commented 3 years ago

@dcharkes Does it have to do with how dart lookup for library ? Is there anyway I could help to speed up the résolution of this issue ? As it’s one of our blocker in an ongoing project.

dcharkes commented 3 years ago

Here are the necessary files need to to replicate https://drive.google.com/file/d/1tpI_StOh7ffPHjVapJJ34dMg2tJka45E/view?usp=sharing

I don't have access to this.

Could you upload it is a GitHub repository instead?

The last time I tried to do static linking for iOS and MacOS I ran into similar problems. I ended up using dynamic libraries instead.

I have succeeded in looking up symbols of a statically linked library by creating the podspec of the framework as the following.

flutter create --platforms=android,ios,macos,windows,linux --template=plugin mylib_staticlib

Modify mylib_staticlib/macos/mylib_staticlib.podspec (for Flutter desktop MacOS):

Pod::Spec.new do |s|
  s.name             = 'mylib_staticlib'
  s.version          = '0.0.1'
  s.summary          = 'A new flutter plugin project.'
  s.description      = <<-DESC
A new flutter plugin project.
                       DESC
  s.homepage         = 'http://example.com'
  s.license          = { :file => '../LICENSE' }
  s.author           = { 'Your Company' => 'email@example.com' }
  s.source           = { :path => '.' }
  s.source_files     = 'Classes/**/*'
  s.dependency 'FlutterMacOS'

  s.platform = :osx, '10.11'
  s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES' }
  s.vendored_libraries = 'Frameworks/libmylib_staticlib.a'
  s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-force_load $(PODS_TARGET_SRCROOT)/Frameworks/libmylib_staticlib.a" }
  s.swift_version = '5.0'
end

And copy the static library in to mylib_staticlib/macos/Frameworks.

For iOS, a similar approach.

mylib_staticlib/ios/mylib_staticlib.podspec:

Pod::Spec.new do |s|
  s.name             = 'mylib_staticlib'
  s.version          = '0.0.1'
  s.summary          = 'A new flutter plugin project.'
  s.description      = <<-DESC
A new flutter plugin project.
                       DESC
  s.homepage         = 'http://example.com'
  s.license          = { :file => '../LICENSE' }
  s.author           = { 'Your Company' => 'email@example.com' }
  s.source           = { :path => '.' }
  s.source_files = 'Classes/**/*'
  s.dependency 'Flutter'
  s.platform = :ios, '8.0'

  # Flutter.framework does not contain a i386 slice.
  s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386' }
  s.vendored_libraries = 'Frameworks/libmylib_staticlib.a'
  s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-force_load $(PODS_TARGET_SRCROOT)/Frameworks/libmylib_staticlib.a" }
  s.swift_version = '5.0'
end

And the fat static library (arm64 for device and x64 for simulator) in mylib_staticlib/ios/Frameworks/libmylib_staticlib.a.

So the 2 important lines in the podspec:

  s.vendored_libraries = 'Frameworks/libmylib_staticlib.a'
  s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-force_load $(PODS_TARGET_SRCROOT)/Frameworks/libmylib_staticlib.a" }

The first statically links the static library, and the second line is the correct invocation to tell the linker to not strip the symbols.

bitsydarel commented 3 years ago

Here's the repository with the necessary files https://github.com/bitsydarel/dart_ffi_static_link_issue

bitsydarel commented 3 years ago

@dcharkes Would be nice to have this test on iOS and on template=app

dcharkes commented 3 years ago

To troubleshoot your problem, we need a complete flutter project, not just the static libraries and header files.

Create a new one with flutter create --platforms=android,ios,macos,windows,linux --template=plugin <insert-name>, copy the static library in to the flutter plugin, modify the podspec, and try to lookup the symbol in a simple flutter app causing the error.

dcharkes commented 3 years ago

@dcharkes Would be nice to have this test on iOS and on template=app

You are always going to need a template=plugin approach here, both for static libraries, dynamic libraries and source code. (Longer explanation: the plugin creates all the necessary build configuration for including native code.)

bitsydarel commented 3 years ago

@dcharkes So basically meaning adding it in an application directly won't do ? is there a reason for the why the app can't create those build configuration and if possible what are those build configuration ?

dcharkes commented 3 years ago

One could possibly. I am not familiar with the how the Flutter app/plugin builds are setup to tell you how. A good starting point for reverse engineering that would be to take a look at what files are generated when doing flutter create --template=plugin. And otherwise ask a question on the Flutter issue tracker.

Any specific reason why you don't want to have a Flutter app + Flutter plugin right next to it in the same repository? And then have a ../myplugin dependency in your pubspec.yaml?

bitsydarel commented 3 years ago

@dcharkes project pushed, also added the lookup functions and on the podspec file also added the required setup as suggested. But still not working on iOS

https://github.com/bitsydarel/dart_ffi_static_link_issue

bitsydarel commented 3 years ago

`[VERBOSE-2:ui_dart_state.cc(199)] Unhandled Exception: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, FPDF_DestroyLibrary): symbol not found)

0 DynamicLibrary.lookup (dart:ffi-patch/ffi_dynamic_library_patch.dart:31:29)

1 new PdfRenderer.load (package:dart_ffii_static_link_issue/src/pdf_renderer.dart:454:12)

2 DartFfiiStaticLinkIssue.checkPDFiumLink (package:dart_ffii_static_link_issue/dart_ffii_static_link_issue.dart:20:46)

3 DartFfiiStaticLinkIssue.platformVersion (package:dart_ffii_static_link_issue/dart_ffii_static_link_issue.dart:14:5)

4 _MyAppState.initPlatformState (package:dart_ffii_static_link_issue_example/main.dart:32:41)

5 _MyAppState.initState (package:dart_ffii_static_link_issue_example/main.dart:22:5)

6 StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:4711:57)

7 ComponentElement.mount (package:flutter/src/widgets/framework.dart:4548:5)

8 Element.inflateWidget (package:flutter/src/widgets/fra<…>`

dcharkes commented 3 years ago

Your reproduction doesn't work here. The static library is missing symbols. Did you mean to include a static library with the standard c/c++ libs as well?

Xcode's output:
↳
    Undefined symbols for architecture arm64:
      "std::__1::basic_streambuf<char, std::__1::char_traits<char> >::setbuf(char*, long)", referenced from:
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cpdf_streamparser.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cfdf_document.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cpdf_syntax_parser.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(fpdf_parser_decode.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cpdf_generateap.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cfx_fontmapper.o)
          vtable for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> > in libpdfium.a(cpdfsdk_appstream.o)

...

full log

bitsydarel commented 3 years ago

@dcharkes if you check the android pdfium folder, you will see other libraries that the library would need

dcharkes commented 3 years ago

@dcharkes if you check the android pdfium folder, you will see other libraries that the library would need

https://github.com/bitsydarel/dart_ffi_static_link_issue/tree/main/pdfium/android

Those are all shared libraries, not static libraries, and not compiled in a way MacOS/iOS can understand them.

Please provide a working reproduction.

paulocoutinhox commented 3 years ago

Hi @bitsydarel,

This is simple. Add on "OTHER_LDFLAGS" the link to "c++" as i showed before in xcode sample.

File: dart_ffii_static_link_issue.podspec

s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-force_load $(PODS_TARGET_SRCROOT)/Frameworks/libpdfium.a -lc++" }
bitsydarel commented 3 years ago

Hi @paulo-coutinho compilation now's working but it's still can't find the symbol `[VERBOSE-2:ui_dart_state.cc(199)] Unhandled Exception: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, FPDF_DestroyLibrary): symbol not found)

0 DynamicLibrary.lookup (dart:ffi-patch/ffi_dynamic_library_patch.dart:31:29)

1 new PdfRenderer.load (package:dart_ffii_static_link_issue/src/pdf_renderer.dart:454:12)

2 DartFfiiStaticLinkIssue.checkPDFiumLink (package:dart_ffii_static_link_issue/dart_ffii_static_link_issue.dart:20:46)

3 DartFfiiStaticLinkIssue.platformVersion (package:dart_ffii_static_link_issue/dart_ffii_static_link_issue.dart:14:5)

4 _MyAppState.initPlatformState (package:dart_ffii_static_link_issue_example/main.dart:32:41)

5 _MyAppState.initState (package:dart_ffii_static_link_issue_example/main.dart:22:5)

6 StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:4711:57)

7 ComponentElement.mount (package:flutter/src/widgets/framework.dart:4548:5)

8 Element.inflateWidget (package:flutter/src/widgets/fra<…>

`

@dcharkes Now the sample is reproducible, i pushed the changes so you could check if it's dart that can't load the symbols.

Thank you guys for the help!

paulocoutinhox commented 3 years ago

How do you testing it?

bitsydarel commented 3 years ago

When you run the app if you check the console you should see logs about about the running app.

paulocoutinhox commented 3 years ago

Hi,

Im investigating the generated binary from the Runner app and Xcode don't link with it:

nm /Users/paulo/Library/Developer/Xcode/DerivedData/Runner-aaxqkqpcwbuqojfckhanjbwlqrbu/Build/Products/Debug-iphoneos/Runner.app/Runner| grep FPDF
[nothing]

And i make the same thing with that my sample using the same library and it is linked correctly:

nm /Users/paulo/Library/Developer/Xcode/DerivedData/PDFiumTest-floybqwuisyrxufsdrwjfipgjiwy/Build/Products/Debug-iphoneos/PDFiumTest.app/PDFiumTest | grep FPDF
0000000100069874 t _FPDFPage_GetCropBox
000000010006978c t _FPDFPage_GetMediaBox
0000000100069318 t _FPDFText_CountChars
0000000100069550 t _FPDFText_CountRects
00000001000695cc t _FPDFText_GetBoundedText
0000000100069358 t _FPDFText_GetCharBox
00000001000693f8 t _FPDFText_GetCharIndexAtPos
0000000100069328 t _FPDFText_GetFontSize
000000010006955c t _FPDFText_GetRect
0000000100069440 t _FPDFText_GetText
0000000100069278 t _FPDFText_LoadPage
000000010006a5b8 t _FPDF_CloseDocument
000000010006a4e0 t _FPDF_ClosePage
000000010006a45c t _FPDF_GetPageBoundingBox
000000010006a2d8 t _FPDF_GetPageCount
000000010006a14c t _FPDF_InitLibraryWithConfig
000000010006a1c4 t _FPDF_LoadDocument
000000010006a310 t _FPDF_LoadPage
000000010006898c t __Z20CPDFPageFromFPDFPageP13fpdf_page_t__
0000000100068980 t __Z20FPDFPageFromIPDFPageP9IPDF_Page
000000010006897c t __Z20IPDFPageFromFPDFPageP13fpdf_page_t__
0000000100068984 t __Z28CPDFDocumentFromFPDFDocumentP17fpdf_document_t__
0000000100068988 t __Z28FPDFDocumentFromCPDFDocumentP13CPDF_Document

The library is not the problem, but the link process.

paulocoutinhox commented 3 years ago

You can use any library name in podspec that it don't will generate error and will open the app:

s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386' }
s.vendored_libraries = 'libpdfiumXYZ.a'
s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-lc++ -lz" }

So you problem is masked.

bitsydarel commented 3 years ago

If your sample was using podspec, how would you link the library ?

paulocoutinhox commented 3 years ago

It is easy. I made it work here:

https://github.com/paulo-coutinho/pdfium-test/pull/1

You need:

cd ios
pod install
open PDFiumTest.xcworkspace
bitsydarel commented 3 years ago

@dcharkes any update because the suggested changes by @paulo-coutinho does actually make the code compile and also make it available in swift or objective c code.

But dart can't see it...

Latest configuration: `s.source_files = 'Classes/*/' s.dependency 'Flutter' s.platform = :ios, '9.0'

s.libraries = ["c++", "z"]

Flutter.framework does not contain a i386 slice.

s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386' } s.vendored_libraries = 'Frameworks/libpdfium.a' s.pod_target_xcconfig = { "OTHER_LDFLAGS" => "-force_load $(PODS_TARGET_SRCROOT)/Frameworks/libpdfium.a" } s.swift_version = '5.0'`

jmgeffroy commented 3 years ago

Hi @bitsydarel,

I am also struggling with the exact same issue, for PDFium too! Did you make any progress since your last message? I'll post here if I find a solution.

Jean-Marie

bitsydarel commented 3 years ago

hi, @jmgeffroy

Waiting on @dcharkes, because it’s seems to be a dart ffi issue as swift and obj can see and use the library.

jmgeffroy commented 3 years ago

Hi @bitsydarel, Thanks a lot for your lightning-fast reply! OK, so I'll eagerly await, too ;-)

dcharkes commented 3 years ago

@bitsydarel I am able to look up symbols fine on iOS/MacOS with a small static library built by CMake using the podspec from https://github.com/dart-lang/sdk/issues/44328#issuecomment-855682903.

I presume either the static library is built in a different way, or your project setup is not the same. I haven't found the time to dig into your repo.

I'll see if I can upload my working repro somewhere so you can work from there.

bitsydarel commented 3 years ago

@dcharkes strangely we tried with other libraries that have dependency with other c/c++ libraries and combined the libraries using lipo.

Dart can’t see the symbols but rust and swift can, is there something different in how dart look for symbols for static libraries ?

dcharkes commented 3 years ago

Dart just uses dlsym on iOS and MacOS, try using dlsym from C or Rust to see if you can see the symbols while trying to dynamically link rather than statically link.

no-response[bot] commented 3 years ago

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away! Thanks for your contribution.

zhengbomo commented 1 year ago

any solution?