search

vgorloff / swift-everywhere-toolchain

Automated workflow to compile Swift Toolchain, for making Android apps with Swift.
MIT License
85 stars 25 forks source link

Where is concurrency lib? #138

Open johnburkey opened 2 years ago

johnburkey commented 2 years ago

Where is _concurrency lib :-)

See discussion section!

Happy Holidays!

vgorloff commented 2 years ago

I wasn't used "new concurrency" on Android yet. For example, in this project - https://github.com/vgorloff/swift-everywhere-samples. I conducted few examples on macOS 12 / Xcode 13.1, but not yet used "new concurrency" in production or commercial apps.

Will try to use few "new concurrency" examples on Android.

johnburkey commented 2 years ago

Thanks!

Happy Holidays !

On Dec 29, 2021, at 10:02 AM, Vlad Gorlov @.***> wrote:

I wasn't used "new concurrency" on Android yet. For example, in this project - https://github.com/vgorloff/swift-everywhere-samples https://github.com/vgorloff/swift-everywhere-samples. I conducted few examples on macOS 12 / Xcode 13.1, but not yet used "new concurrency" in production or commercial apps.

Will try to use few "new concurrency" examples on Android.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1002664585, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PMCVFAWMUERQZ2XWOTUTMWK5ANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

johnburkey commented 2 years ago

Oh, and we use the new concurrency on iOS/macOS/Linux and its good now for normal operations.

On Dec 29, 2021, at 1:15 PM, John Burkey @.***> wrote:

Thanks!

Happy Holidays !

On Dec 29, 2021, at 10:02 AM, Vlad Gorlov @. @.>> wrote:

I wasn't used "new concurrency" on Android yet. For example, in this project - https://github.com/vgorloff/swift-everywhere-samples https://github.com/vgorloff/swift-everywhere-samples. I conducted few examples on macOS 12 / Xcode 13.1, but not yet used "new concurrency" in production or commercial apps.

Will try to use few "new concurrency" examples on Android.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1002664585, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PMCVFAWMUERQZ2XWOTUTMWK5ANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

Building Concurrency example (https://github.com/vgorloff/swift-everywhere-toolchain/commit/645e75d91bcb423a6b349678820bf3cf756a6ebb) gives me the following error:

{0}vova@MacBook-Pro:sample-package$ swift-build-arm-linux-androideabi
<unknown>:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found
<unknown>:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found
<unknown>:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found
/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/Tests/sample-package/Sources/SAConcurrency/SAConcurrencyMain.swift:18:7: error: cannot find 'Task' in scope
      Task {
      ^~~~

Seems the Concurrencyfeature wasn't build when building Android toolchain.

vgorloff commented 2 years ago

The Concurrency feature wasn't enabled during build.

# File CMakeCache.txt
//Enable build of the Swift concurrency module
SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY:BOOL=OFF
vgorloff commented 2 years ago

There is a chicken-egg problem "dispatch needs stdlib" and, at the same time, "stdlib needs dispatch".

# File: stdlib/public/Concurrency/CMakeLists.txt

if(SWIFT_CONCURRENCY_USES_DISPATCH)
  if(NOT CMAKE_SYSTEM_NAME STREQUAL Darwin)
    include_directories(AFTER
                          ${SWIFT_PATH_TO_LIBDISPATCH_SOURCE})

    # FIXME: we can't rely on libdispatch having been built for the
    # target at this point in the process.  Currently, we're relying
    # on soft-linking.
    list(APPEND swift_concurrency_link_libraries
      dispatch)
  endif()
endif()
vgorloff commented 2 years ago

To build Concurrency feature the Dispatch subproject needs to be build during stdlib build. It is configured via cmake/modules/Libdispatch.cmake.

vgorloff commented 2 years ago

Dispatch subproject fails to build due compiler flag -Werror :)

johnburkey commented 2 years ago

:-)

Sent from my iPhone

On Jan 1, 2022, at 8:40 AM, Vlad Gorlov @.***> wrote:

 Dispatch subproject fails to build due compiler flag -Werror :)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

johnburkey commented 2 years ago

does it build?

vgorloff commented 2 years ago

I will try to figure out in coming days how to pass arguments to external project (dispatch in our case). After quick search I didn't managed to find "who" is passing that -Werror flag. Maybe it is a CMake default.

johnburkey commented 2 years ago

Maybe ping compnerd on slack channel for swift on android ? He’s awesome and very aquatinted with stuff like this as he’s the windows build champ on top of all of his other magic.

On Jan 5, 2022, at 2:50 AM, Vlad Gorlov @.***> wrote:

 I will try to figure out in coming days how to pass arguments to external project (dispatch in our case). After quick search I didn't managed to find "who" is passing that -Werror flag. Maybe it is a CMake default.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

johnburkey commented 2 years ago

https://join.slack.com/t/swift-android/shared_invite/zt-11grcjtps-UHeM3etApuYdayxAKv436A

johnburkey commented 2 years ago

Sounds like they want to do a strategy discussion with you too - Nice folks .

vgorloff commented 2 years ago

Building of "libdispatch" is not a problem. There is already builder for it (https://github.com/vgorloff/swift-everywhere-toolchain/blob/master/lib/Builders/DispatchBuilder.js). Issue that CMake is a functional-programming thing which consumes time to figure out needed options to pass :)

johnburkey commented 2 years ago

Compnerd knows the swift builds super well just thinking he could save you time ;-)

On Jan 5, 2022, at 4:14 PM, Vlad Gorlov @.***> wrote:

 Building of "libdispatch" is not a problem. There is already builder for it (https://github.com/vgorloff/swift-everywhere-toolchain/blob/master/lib/Builders/DispatchBuilder.js). Issue that CMake is a functional-programming thing which consumes time to figure out needed options to pass :)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

Seems "libDispatch" itself defined compiler flags here: https://github.com/apple/swift-corelibs-libdispatch/blob/swift-5.5.2-RELEASE/cmake/modules/DispatchCompilerWarnings.cmake

Strange why I don't have this issue when building libdispatch standalone. Maybe because i am passing C/CXX flags explicitly.

vgorloff commented 2 years ago

The issue with -Werror is fixed by adding code below:

    if(CMAKE_HOST_SYSTEM_NAME STREQUAL Darwin)
      add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-unused-command-line-argument>)
    endif()

The next error seems due minSDKVersion 21 (https://stackoverflow.com/a/53217341/1418981):

/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Sources/swift-corelibs-libdispatch/src/event/event_epoll.c:460:33: error: use of undeclared identifier 'EPOLL_CLOEXEC'
        _dispatch_epfd = epoll_create1(EPOLL_CLOEXEC);
                                       ^
1 error generated.

Cmake uses SDK 16 by default:

-- ANDROID_PLATFORM not set. Defaulting to minimum supported version
16.
vgorloff commented 2 years ago

Passing -DANDROID_PLATFORM=${ANDROID_PLATFORM} in cmake/modules/Libdispatch.cmake helped to build libswift_Concurrency.so for armv7.

Next error when building for aarch64:

-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang
-- Check for working C compiler: /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang - broken
CMake Error at /usr/local/Cellar/cmake/3.22.0/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"

  is not able to compile a simple test program.

Configure is not finished successfully :0

vgorloff commented 2 years ago

When I am building libdispatch as standalone, the CMAKE_C_FLAGS defined as empty string. When building libdispatch via stdlib, then CMAKE_C_FLAGS defined as CMAKE_C_FLAGS:STRING=-g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wno-comment -fdiagnostics-color

Thus the following configure error seems due unneeded flag -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21

-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang
-- Check for working C compiler: /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang - broken
CMake Error at /usr/local/Cellar/cmake/3.22.0/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/libdispatch-android-aarch64-prefix/src/libdispatch-android-aarch64-build/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/local/bin/ninja cmTC_06ead && [1/2] Building C object CMakeFiles/cmTC_06ead.dir/testCCompiler.c.o
    clang: warning: argument unused during compilation: '-L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21' [-Wunused-command-line-argument]
    [2/2] Linking C executable cmTC_06ead
    FAILED: cmTC_06ead
    : && /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang --target=armv7-none-linux-androideabi21 --gcc-toolchain=/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64 --sysroot=/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security  -L`/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21` -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wno-comment -fdiagnostics-color -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libatomic.a -static-libstdc++ -Wl,--build-id -Wl,--fatal-warnings -Wl,--exclude-libs,libunwind.a -Wl,--no-undefined -Qunused-arguments -Wl,--gc-sections CMakeFiles/cmTC_06ead.dir/testCCompiler.c.o -o cmTC_06ead  -latomic -lm && :
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libm.so while searching for m
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libdl.so while searching for dl
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libdl.so while searching for dl
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libc.so while searching for c
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libdl.so while searching for dl
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: warning: skipping incompatible /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libdl.so while searching for dl
    /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: treating warnings as errors
vgorloff commented 2 years ago

When building stdlib, I am passing manually the C flag -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21. And then Cmake picks it and passes to the libdispatch subproject. Maybe I just need not to pass it.

vgorloff commented 2 years ago

Next error:

FAILED: lib/swift/android/aarch64/libswift_Concurrency.so
: && /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang --target=aarch64-none-linux-android21 --gcc-toolchain=/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64 --sysroot=/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security  -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/21 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wno-comment -fdiagnostics-color -O2  -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libatomic.a -static-libstdc++ -Wl,--build-id -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments     -shared -Wl,-soname,libswift_Concurrency.so -target aarch64-unknown-linux-android21 --sysroot=/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -B /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin -lm -fuse-ld=gold -shared  -o lib/swift/android/aarch64/libswift_Concurrency.so stdlib/public/Concurrency/ANDROID/aarch64/_Concurrency.o lib/swift/android/aarch64/swiftrt.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/__/CompatibilityOverride/CompatibilityOverride.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/Actor.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/AsyncLet.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/Error.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/LinkCompatibilityShims.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/Task.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/TaskAlloc.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/TaskStatus.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/TaskGroup.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/TaskLocal.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/ThreadSanitizer.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/Mutex.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/AsyncStream.cpp.o stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/linker-support/magic-symbols-for-install-name.c.o -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-host/llvm-project/./lib   -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/./lib/swift/android/aarch64   -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-host/swift/bin/../lib/swift/android/aarch64   -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-host/swift/bin/../lib/swift/android   -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/swift   -L/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/../lib/gcc/aarch64-linux-android/4.9.x   -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Install/darwin-aarch64/icu/lib   -L/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/libdispatch-android-aarch64-prefix/lib -ldispatch  -ldl  -llog  /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++abi.a  /Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so  /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Install/darwin-aarch64/icu/lib/libicui18nswift.so  /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Install/darwin-aarch64/icu/lib/libicuucswift.so  lib/swift/android/aarch64/libswiftGlibc.so  lib/swift/android/aarch64/libswiftCore.so  -latomic -lm && :
/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin/ld.gold: warning: skipping incompatible /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/./lib/swift/android/aarch64/libdispatch.so while searching for dispatch
/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin/ld.gold: warning: skipping incompatible /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/libdispatch-android-aarch64-prefix/lib/libdispatch.so while searching for dispatch
/Volumes/Shared/Data/Android/sdk/ndk/21.4.7075529/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin/ld.gold: error: cannot find -ldispatch
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to 'dispatch_queue_attr_make_with_qos_class'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to 'dispatch_queue_create'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to 'dispatch_queue_set_width'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalImpl(swift::Job*): error: undefined reference to 'dispatch_release'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_time'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_after_f'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to '_dispatch_queue_attr_concurrent'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_queue_attr_make_with_qos_class'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_queue_create'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_queue_set_width'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueGlobalWithDelayImpl(unsigned long long, swift::Job*): error: undefined reference to 'dispatch_release'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueMainExecutor: error: undefined reference to '_dispatch_main_q'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueMainExecutor: error: undefined reference to '_dispatch_main_q'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueMainExecutorImpl(swift::Job*): error: undefined reference to '_dispatch_main_q'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function swift_task_enqueueMainExecutorImpl(swift::Job*): error: undefined reference to '_dispatch_main_q'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/GlobalExecutor.cpp.o:GlobalExecutor.cpp:function dispatchEnqueueDispatchAsync(dispatch_queue_s*, void*, unsigned int): error: undefined reference to 'dispatch_async_f'
stdlib/public/Concurrency/CMakeFiles/swift_Concurrency-android-aarch64.dir/Task.cpp.o:Task.cpp:function swift_task_asyncMainDrainQueueImpl(): error: undefined reference to 'dispatch_main'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Obviously because libdispatch for some reason was built for armv7 instead for aarch64 :)

{0}vova@MacBook-Pro:swift-everywhere-toolchain$ file /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/lib/swift/android/aarch64/libswiftCore.so
/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/lib/swift/android/aarch64/libswiftCore.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=994cfd8863a2c520eb9e21ff8538133e63c478ca, with debug_info, not stripped
{0}vova@MacBook-Pro:swift-everywhere-toolchain$ file /Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/lib/swift/android/aarch64/libdispatch.so
/Volumes/Shared/Git/MyProjects/swift-everywhere-toolchain/ToolChain/Build/darwin-aarch64/swift-stdlib/lib/swift/android/aarch64/libdispatch.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=b8dfcdb8a04a347ce81569ad7dcec4457acf3e8f, with debug_info, not stripped
{0}vova@MacBook-Pro:swift-everywhere-toolchain$
vgorloff commented 2 years ago

For some reason the option CMAKE_SYSTEM_PROCESSOR set as armv7-a in file /.../CMakeFiles/3.22.0/CMakeSystem.cmake for libdispatch subproject. At the same time stdlib itself has value for the same option set as aarch64. Which is correct one.

vgorloff commented 2 years ago

The above issue fixed by passing Cmake flag -DANDROID_ABI=${ANDROID_ABI} to the libdispatch subproject.

vgorloff commented 2 years ago

Ok. I am able to build stdlib with libswift_Concurrency.so for all 4 Android archs: arm 32/64 bit and x86 32/64 bit.

Now it is time to make clean rebuild of Swift, stdlib, libDispatch, libFoundation and then try Async/Await example again.

johnburkey commented 2 years ago

Yay!

On Jan 6, 2022, at 4:13 AM, Vlad Gorlov @.***> wrote:

 Ok. I am able to build stdlib with libswift_Concurrency.so for all 4 Android archs: arm 32/64 bit and x86 32/64 bit.

Now it is time to make clean rebuild of Swift, stdlib, libDispatch, libFoundation and then try Async/Await example again.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

But sample code which works on macOS crashes with Segmentation fault on Android :0

johnburkey commented 2 years ago

Getting closer!

On Jan 6, 2022, at 8:55 AM, Vlad Gorlov @.***> wrote:

But sample code which works on macOS crashes with Segmentation fault on Android :0

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1006654303, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PIP2MVFZVD6ODCD4J3UUWUONANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

I will try one idea, but if it won't work, then I will come back to troubleshooting after next major release of Swift - Swift 5.5.3 or Swift 5.6.0.

The Swift Concurrency is experimental thing. Even Cmake option, which is used to activate this feature, defined as following -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=TRUE/FALSE

johnburkey commented 2 years ago

I think that flag is the old flag when it was experimental - i haven’t had any issues with the feature since they fixed the bugs just post 5.5.

I’m sure compnerd has good advice on this - find him in the slack channel ?

On Jan 6, 2022, at 9:11 AM, Vlad Gorlov @.***> wrote:

 I will try one idea, but if it won't work, then I will come back to troubleshooting after next major release of Swift - Swift 5.5.3 or Swift 5.6.0.

The Swift Concurrency is experimental thing. Even Cmake option, which is used to activate this feature, defined as following -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=TRUE/FALSE

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

The release which builds concurrency is ready https://github.com/vgorloff/swift-everywhere-toolchain/releases/tag/1.0.78 But on runtime new Swift concurrency seems will alway crash.

vgorloff commented 2 years ago

Here is what adb logcat shows me:

01-06 19:52:55.058  4489  4497 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0x18 in tid 4497 (Exe), pid 4489 (Exe)
01-06 19:52:55.097  4501  4501 W crash_dump32: type=1400 audit(0.0:2863): avc: denied { search } for name="tmp" dev="dm-2" ino=130818 scontext=u:r:crash_dump:s0 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=0
01-06 19:52:55.100  4501  4501 I chatty  : uid=10110(com.whatsapp) crash_dump32 identical 14 lines
01-06 19:52:55.100  4501  4501 W crash_dump32: type=1400 audit(0.0:2878): avc: denied { search } for name="tmp" dev="dm-2" ino=130818 scontext=u:r:crash_dump:s0 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=0
01-06 19:52:55.107  4501  4501 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone
01-06 19:52:55.108   587   587 I /system/bin/tombstoned: received crash request for pid 4489
01-06 19:52:55.110  4501  4501 I crash_dump32: performing dump of process 4489 (target tid = 4497)
01-06 19:52:55.110  4501  4501 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
01-06 19:52:55.110  4501  4501 W crash_dump32: type=1400 audit(0.0:2879): avc: denied { search } for name="tmp" dev="dm-2" ino=130818 scontext=u:r:crash_dump:s0 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=0
01-06 19:52:55.111  4501  4501 F DEBUG   : Build fingerprint: 'google/bullhead/bullhead:8.1.0/OPM7.181205.001/5080180:user/release-keys'
01-06 19:52:55.111  4501  4501 F DEBUG   : Revision: 'rev_1.0'
01-06 19:52:55.111  4501  4501 F DEBUG   : ABI: 'arm'
01-06 19:52:55.111  4501  4501 F DEBUG   : pid: 4489, tid: 4497, name: Exe  >>> /data/local/tmp/sample-Exe/Exe <<<
01-06 19:52:55.111  4501  4501 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x18
01-06 19:52:55.111  4501  4501 F DEBUG   : Cause: null pointer dereference
01-06 19:52:55.111  4501  4501 F DEBUG   :     r0 00000000  r1 6a205680  r2 00000019  r3 80000000
01-06 19:52:55.111  4501  4501 F DEBUG   :     r4 00000000  r5 00000000  r6 f4f8b1cb  r7 00000000
01-06 19:52:55.111  4501  4501 F DEBUG   :     r8 f38a2aa0  r9 02001900  sl f34400f0  fp f38bd740
01-06 19:52:55.111  4501  4501 F DEBUG   :     ip f4fa6fbc  sp f327f838  lr f4f8a51b  pc 00000018  cpsr 80070030
01-06 19:52:55.113  4501  4501 F DEBUG   : 
01-06 19:52:55.113  4501  4501 F DEBUG   : backtrace:
01-06 19:52:55.114  4501  4501 F DEBUG   :     #00 pc 00000018  <unknown>
01-06 19:52:55.115  4501  4501 F DEBUG   :     #01 pc 00043517  /data/local/tmp/sample-Exe/libswift_Concurrency.so
01-06 19:52:55.115  4501  4501 F DEBUG   :     #02 pc fffffffd  <unknown>
01-06 19:52:55.186   763  1089 W NativeCrashListener: Couldn't find ProcessRecord for pid 4489
01-06 19:52:55.143  4501  4501 I chatty  : uid=10110(com.whatsapp) identical 3 lines
01-06 19:52:55.180  4501  4501 W crash_dump32: type=1400 audit(0.0:2883): avc: denied { search } for name="tmp" dev="dm-2" ino=130818 scontext=u:r:crash_dump:s0 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=0
01-06 19:52:55.188   587   587 W /system/bin/tombstoned: crash socket received short read of length 0 (expected 12)
01-06 19:52:55.190   763   784 I BootReceiver: Copying /data/tombstones/tombstone_06 to DropBox (SYSTEM_TOMBSTONE)
01-06 19:52:55.216   763   777 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.stats.service.DropBoxEntryAddedReceiver
01-06 19:52:55.216   763   777 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.chimera.GmsIntentOperationService$PersistentTrustedReceiver

The issue is null pointer dereference.

johnburkey commented 2 years ago

Got symbols ?

vgorloff commented 2 years ago

Not too much: <anonymous:f1d90000> instead of <unknown> :)

01-06 20:29:25.735  5762  5762 F DEBUG   : backtrace:
01-06 20:29:25.735  5762  5762 F DEBUG   :     #00 pc 000fb970  <anonymous:f1d90000>
01-06 20:29:25.735  5762  5762 F DEBUG   :     #01 pc 000834c7  /data/local/tmp/sample-Exe/libswift_Concurrency.so
01-06 20:29:25.735  5762  5762 F DEBUG   :     #02 pc 219021ac  <unknown>
vgorloff commented 2 years ago

The Concurrency actually working with latest toolchain, but for macOS and Android it works differently.

public class SAConcurrencyMain {

   private var task: Task<Void, Error>?

   public init() {
      print("[SAC] Concurrency: Init")
   }

   public func execute(completion: @escaping () -> Void) {
      print("[SAC] Concurrency: Start")
#if !false
      task = Task.detached(priority: .high) {
         let urls = await self.getURLs()
         print("[SAC] Concurrency: Got \(urls.count) urls.")
         print("[SAC] Concurrency: End")
         completion()
      }
#else // Below works on macOS but crashes on Android. See: https://github.com/vgorloff/swift-everywhere-toolchain/issues/138
      Task {
         let task = Task.detached(priority: .userInitiated) { () -> Int in
            let urls = await self.getURLs()
            return urls.count
         }
         let value = await task.value
         print("[SAC] Concurrency: Got \(value) urls.")
         print("[SAC] Concurrency: End")
         completion()
      }
#endif
   }

   private func getURLs() async -> [String] {
      Thread.sleep(forTimeInterval: 1)
      return ["https://docs.swift.org/", "https://google.com/", "https://ibm.com/"]
   }

}

The Task inside a Task not working on Android, but works on macOS:

Task {
         let task = Task.detached(priority: .userInitiated) { () -> Int in
             ...
         }
         let value = await task.value
        ...
}
johnburkey commented 2 years ago

So simple tasks work on droid but complicated ones (task inside task) crashes ?

On Jan 6, 2022, at 4:33 PM, Vlad Gorlov @.***> wrote:

 The Concurrency actually working with latest toolchain, but for macOS and Android it works differently.

public class SAConcurrencyMain {

private var task: Task<Void, Error>?

public init() { print("[SAC] Concurrency: Init") }

public func execute(completion: @escaping () -> Void) { print("[SAC] Concurrency: Start")

if !false

  task = Task.detached(priority: .high) {
     let urls = await self.getURLs()
     print("[SAC] Concurrency: Got \(urls.count) urls.")
     print("[SAC] Concurrency: End")
     completion()
  }

else // Below works on macOS but crashes on Android. See: https://github.com/vgorloff/swift-everywhere-toolchain/issues/138

  Task {
     let task = Task.detached(priority: .userInitiated) { () -> Int in
        let urls = await self.getURLs()
        return urls.count
     }
     let value = await task.value
     print("[SAC] Concurrency: Got \(value) urls.")
     print("[SAC] Concurrency: End")
     completion()
  }

endif

}

private func getURLs() async -> [String] { Thread.sleep(forTimeInterval: 1) return ["https://docs.swift.org/", "https://google.com/", "https://ibm.com/"] }

} The Task inside a Task not working on Android, but works on macOS:

Task { let task = Task.detached(priority: .userInitiated) { () -> Int in ... } let value = await task.value ... } — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

True. Some pieces of Concurrency seems working on Android as on macOS. Others seems not. Maybe some kind of execution environment on Android is not fully initialised/configured (e.g. MainActor or similar).

johnburkey commented 2 years ago

Let’s find the Linux concurrency people

On Jan 7, 2022, at 2:29 AM, Vlad Gorlov @.***> wrote:

 True. Some pieces of Concurrency seems working on Android as on macOS. Others seems not. Maybe some kind of execution environment on Android is not fully initialised/configured (e.g. MainActor or similar).

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

finagolfin commented 2 years ago

Hmm, looks like you're building the stdlib with the NDK clang, you have to make sure the Concurrency files are built with Apple's fork of clang.

johnburkey commented 2 years ago

Hey! Thanks for this ! Can we jump on that slack channel on swift for android for higher bandwidth? Gorloff you game? I think Vlad is in Europe somewhere, so hoping to catch this "everyone is awake" window.. You don't really need me, i'm merely the pot stirrer. :-)

johnburkey commented 2 years ago

Buttaface, any stability issues once you had it running? Saleem mentioned to me that he had to work over the libdispatch on windows stuff to get things stable.

finagolfin commented 2 years ago

I'm not sure he's awake. 😉 I don't think there will be much to discuss, that is the only issue I've had with getting it working.

I can't really speak to stability, as I don't use the new Concurrency runtime on Android much. However, I run the full compiler validation testsuite on Android with the latest trunk tag once a month, and all the Concurrency tests pass.

johnburkey commented 2 years ago

Wake up VLAD ! :-)

Well hoping to have a powwow as needed to get him to a good place. Then later we can all talk about how we can get assistance getting things you guys need into main?

I can imagine thats “fun”.

We currently use “course grained” actors enough that we should shake any first things loose.

On Jan 7, 2022, at 11:25 AM, buttaface @.***> wrote:

I'm not sure he's awake. 😉 I don't think there will be much to discuss, that is the only issue I've had with getting it working.

I can't really speak to stability, as I don't use the new Concurrency runtime on Android much. However, I run the full compiler validation testsuite on Android with the latest trunk tag once a month, and all the tests pass.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1007590900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PJPZUMESCQF3KVPMY3UU4O2FANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

finagolfin commented 2 years ago

I think the issue is only because he doesn't use the official build-script, whereas I'm now down to a couple small patches since I use that. There's not much left to upstream for me, just submitting the last handful of patches.

If you run into any issues with Concurrency on Android, just post it in the forums or the bug tracker and we'll take a look.

johnburkey commented 2 years ago

Butta are you on the slack for swift for android ? Higher bandwidth !

On Jan 7, 2022, at 11:47 AM, buttaface @.***> wrote:

I think the issue is only because he doesn't use the official build-script, whereas I'm now down to a couple small patches since I use that. There's not much left to upstream for me, just submitting the last handful of patches https://github.com/search?q=org%3Aapple+author%3Abuttaface&state=open&type=Issues.

If you run into any issues with Concurrency on Android, just post it in the forums or the bug tracker and we'll take a look.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1007604754, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PLVWR7MJRMM5WNHV7LUU4RKHANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

vgorloff commented 2 years ago

Interesting. I didn't know that LLVM has a new flag, which seems emits something into binary which seems is needed to support Concurrency feature.

johnburkey commented 2 years ago

Can we all jump to slack for a couple minutes?

Butta you on that one?

swift-android.slack.com

On Jan 7, 2022, at 11:50 AM, Vlad Gorlov @.***> wrote:

Interesting. I didn't know that LLVM has a new flag, which seems emits something into binary which seems is needed to support Concurrency feature.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1007606840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PMHBKJPMIF4TWFKTC3UU4RWVANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

finagolfin commented 2 years ago

I didn't know about that one. Vlad and I occasionally chat on another Slack: he can find me on there now if he wants.

johnburkey commented 2 years ago

Which slack, can I jump there too? I promise to be nice.

On Jan 7, 2022, at 11:58 AM, buttaface @.***> wrote:

I didn't know about that one. Vlad and I occasionally chat on another Slack: he can find me on there now if he wants.

— Reply to this email directly, view it on GitHub https://github.com/vgorloff/swift-everywhere-toolchain/issues/138#issuecomment-1007612666, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5F3PN4NLCSTGNJFAPXEU3UU4SUNANCNFSM5K57HZOQ. You are receiving this because you authored the thread.

finagolfin commented 2 years ago

I don't think we have invites or are even supposed to publicly talk about it yet. As you said, it's mostly me and him that might need to talk, so we'll get to it whenever. If you want us to look at something in particular, just post it here.