search

flutter-tizen / engine

The Flutter engine
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
6 stars 19 forks source link

Build and release our own gen_snapshots in the engine CI #224

Closed bbrto21 closed 2 years ago

bbrto21 commented 2 years ago

Originally posted by @swift-kim in https://github.com/flutter-tizen/engine/issues/219#issuecomment-992206908

Target host

swift-kim commented 2 years ago

https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment

The problematic part seems to be this part:

  • On Windows:
    • Visual Studio 2017 or later (required for non-Googlers only).
    • Windows 10 SDK (required for non-Googlers only).
    • Be sure to install the "Debugging Tools for Windows" feature.

Can these prerequisites be installed from the command line on GitHub-hosted instances?

bbrto21 commented 2 years ago
  • Visual Studio 2017 or later (required for non-Googlers only).

Visual Studio 2022 was installed on my Windows machine, but the tool said it is not a supported version :(

bwikbs commented 2 years ago

Can these prerequisites be installed from the command line on GitHub-hosted instances?

It seems to be difficult to get instance what is satisfied that requirement.... May be self hosted runner?

swift-kim commented 2 years ago

Build commands for Windows:

# Targeting tizen-arm-release.
python .\flutter\tools\gn --target-os=linux --linux-cpu=arm --no-goma --runtime-mode=release
ninja -C .\out\linux_release_arm gen_snapshot
WonyoungChoi commented 2 years ago

Github hosted runner provides VS environment 🎉 https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#visual-studio-enterprise-2019

bwikbs commented 2 years ago

Github hosted runner provides VS environment 🎉 https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#visual-studio-enterprise-2019

It's amazing!! github is really generous!

swift-kim commented 2 years ago

The tool side work is done by https://github.com/flutter-tizen/flutter-tizen/pull/296. Now the only work to do is to add gen_snapshot executable for each platform in each of our zipped artifacts (tizen-arm-release.zip and etc.).

e.g. tizen-arm-release.zip should contain:

The better design should be downloading only files that are relevant for the host platform (e.g. only linux-x64 but no darwin-x64 on Linux). How can we accomplish this?

swift-kim commented 2 years ago

@WonyoungChoi

the artifacts will be separated to download only the artifact required for the host platform

How this will be done is exactly what I was asking in the above comment. Ideally, gen_snapshot should be placed in the corresponding engine artifact directory as follows:

$ tree flutter/bin/cache/artifacts/engine/tizen-*-release
flutter/bin/cache/artifacts/engine/tizen-arm-release
├── libflutter_engine.so
├── libflutter_tizen_common.so
├── libflutter_tizen_mobile.so
├── libflutter_tizen_tv.so
├── libflutter_tizen_wearable.so
└── linux-x64
    └── gen_snapshot
flutter/bin/cache/artifacts/engine/tizen-arm64-release
├── libflutter_engine.so
├── libflutter_tizen_common.so
├── libflutter_tizen_mobile.so
├── libflutter_tizen_tv.so
├── libflutter_tizen_wearable.so
└── linux-x64
    └── gen_snapshot

Are you thinking about zipping linux-x64/gen_snapshot separately from tizen-arm-release.zip? What name will the zip file be?

WonyoungChoi commented 2 years ago

In the case of Android, the artifact name (or path) is android-arm-release/windows-x64.zip. maybe we should use the name like tizen-arm-release-windows-x64.zip because Github CI does not support the directory path for the artifact.

swift-kim commented 2 years ago

Now the default engine branch is flutter-2.8.1-tizen. I'll discard the 2.8.0 pre-release when the 2.8.1 artifacts are ready.

swift-kim commented 2 years ago

tizen-arm-release-windows-x64.zip

@WonyoungChoi What files will be included in this archive?

WonyoungChoi commented 2 years ago

tizen-arm-release-windows-x64.zip

@WonyoungChoi What files will be included in this archive?

I think only the gen_snapshot binary. 

tizen-arm-release-windows-x64.zip
==>
tizen-arm-release/
└── windows-x64
    └── gen_snapshot
swift-kim commented 2 years ago

If we have both tizen-arm-release.zip and tizen-arm-release-windows-x64.zip which contain engine binaries and gen_snapshot binary respectively, and the two archives are downloaded in sequence, only the gen_snapshot binary will exist in the final tizen-arm-release directory (see the ArtifactUpdater._downloadArchive implementation).

So it should be better to let tizen-arm-release_windows-x64.zip contain all 6 files:

tizen-arm-release
├── libflutter_engine.so
├── libflutter_tizen_common.so
├── libflutter_tizen_mobile.so
├── libflutter_tizen_tv.so
├── libflutter_tizen_wearable.so
└── windows-x64
    └── gen_snapshot
swift-kim commented 2 years ago

TizenEngineArtifacts.getBinaryDirs will look like this:

List<List<String>> getBinaryDirs() => <List<String>>[
      <String>['tizen-common', 'tizen-common.zip'],
      <String>['tizen-x86-debug', 'tizen-x86-debug.zip'],
      <String>['tizen-arm-debug', 'tizen-arm-debug.zip'],
      <String>['tizen-arm-profile', 'tizen-arm-profile_$hostPlatform.zip'],
      <String>['tizen-arm-release', 'tizen-arm-release_$hostPlatform.zip'],
      <String>['tizen-arm64-debug', 'tizen-arm64-debug.zip'],
      <String>['tizen-arm64-profile', 'tizen-arm64-profile_$hostPlatform.zip'],
      <String>['tizen-arm64-release','tizen-arm64-release_$hostPlatform.zip'],
    ];
WonyoungChoi commented 2 years ago

@swift-kim I want to avoid building the entire engine source on Windows or MacOS runner as much as possible. Those runners will be used for generating only gen_snapshot binaries for each host-platform. How about referring to AndroidGenSnapshotArtifacts?

      <String>['tizen-arm-profile/$hostPlatform', 'tizen-arm-profile_$hostPlatform.zip'],
      <String>['tizen-arm-release/$hostPlatform', 'tizen-arm-release_$hostPlatform.zip'],
      <String>['tizen-arm64-profile/$hostPlatform', 'tizen-arm64-profile_$hostPlatform.zip'],
      <String>['tizen-arm64-release/$hostPlatform','tizen-arm64-release_$hostPlatform.zip'],
swift-kim commented 2 years ago

Trying macOS build...

diff --git a/tools/gn b/tools/gn
index f10bd92221..9ee74ced76 100755
--- a/tools/gn
+++ b/tools/gn
@@ -301,7 +301,7 @@ def to_gn_args(args):
       # attributes in release modes till the toolchain is updated.
       gn_args['skia_enable_api_available_macro'] = args.runtime_mode != "release"

-    if sys.platform == 'darwin' and args.target_os not in ['android', 'fuchsia']:
+    if sys.platform == 'darwin' and args.target_os not in ['android', 'fuchsia', 'linux']:
       # OpenGL is deprecated on macOS > 10.11.
       # This is not necessarily needed but enabling this until we have a way to
       # build a macOS metal only shell and a gl only shell.
diff --git a/build/config/sysroot.gni b/build/config/sysroot.gni
index 17ffd1b..9e985e7 100644
--- a/build/config/sysroot.gni
+++ b/build/config/sysroot.gni
@@ -14,7 +14,7 @@ declare_args() {

   # Whether to use the default sysroot when building for Linux, if an explicit
   # sysroot isn't set.
-  use_default_linux_sysroot = true
+  use_default_linux_sysroot = false

   # The absolute path to the Xcode toolchain. This is used to look for headers
   # that usually ship with the toolchain like c++/v1.
flutter/tools/gn \
--target-os linux \
--linux-cpu arm \
--no-goma \
--target-toolchain `pwd`/tizen_tools/toolchains \
--target-sysroot `pwd`/tizen_tools/sysroot/arm \
--target-triple armv7l-tizen-linux-gnueabi \
--runtime-mode release
ninja -C out/linux_release_arm gen_snapshot

Build succeeded but the resulting gen_snapshot is not a host executable but an arm executable.

swift@Sangwookui-MacBookAir src % file out/linux_release_arm/gen_snapshot
out/linux_release_arm/gen_snapshot: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, BuildID[sha1]=1c9505aa0670d319ad765bbb05c6edb0a8b097db, for GNU/Linux 3.2.0, stripped

~Still need to investigate why.~

Hmm. Maybe the directory out/linux_release_arm/clang_x64 is not generated because I'm on an arm64 Mac. (Instead clang_arm64 is generated.) I need to test on an Intel Mac.

WonyoungChoi commented 2 years ago

@swift-kim I tried to build on my Intel Mac, but the gen_snapshot is not generated in out/linux_release_arm/clang_x64 also. 😢

WonyoungChoi commented 2 years ago

MacOS build successful! 😃 

diff --git a/build/config/BUILDCONFIG.gn b/build/config/BUILDCONFIG.gn
index 0f8eb93..613aa83 100644
--- a/build/config/BUILDCONFIG.gn
+++ b/build/config/BUILDCONFIG.gn
@@ -536,6 +536,14 @@ if (custom_toolchain != "") {
   if (is_chromeos && cros_use_custom_toolchain) {
     set_default_toolchain("//build/toolchain/cros:target")
   }
+  if (host_os == "mac") {
+    host_toolchain = "//build/toolchain/mac:clang_$host_cpu"
+    if (current_cpu == "arm") {
+      set_default_toolchain("//build/toolchain/linux:$current_cpu")
+    } else {
+      set_default_toolchain("//build/toolchain/linux:clang_$current_cpu")
+    }
+  }
 } else if (is_mac) {
   host_toolchain = "//build/toolchain/mac:clang_$host_cpu"
   set_default_toolchain("//build/toolchain/mac:clang_$current_cpu")
diff --git a/tools/gn b/tools/gn
index f10bd92..e7482a7 100755
--- a/tools/gn
+++ b/tools/gn
@@ -390,6 +390,13 @@ def to_gn_args(args):
     if args.target_os in ['android'] and gn_args['target_cpu'] in ['x64' , 'arm64']:
       gn_args['dart_use_compressed_pointers'] = True

+    # Overwrite arguments for building tizen files on MacOS.
+    if args.macos_build_tizen:
+      gn_args['allow_deprecated_api_calls'] = False
+      gn_args['skia_use_metal'] = False
+      gn_args['shell_enable_metal'] = False
+      gn_args['use_default_linux_sysroot'] = False
+
     return gn_args

 def parse_args(args):
@@ -496,6 +503,8 @@ def parse_args(args):
                       'true and is enabled on CI.')
   parser.add_argument('--no-prebuilt-dart-sdk', dest='prebuilt_dart_sdk', action='store_false')

+  parser.add_argument('--macos-build-tizen', default=False, action='store_true')
+
   # Sanitizers.
   parser.add_argument('--asan', default=False, action='store_true')
   parser.add_argument('--lsan', default=False, action='store_true')
src % ./flutter/tools/gn --linux --linux-cpu arm --runtime-mode profile --no-goma --macos-build-tizen
src % ninja -C out/linux_profile_arm clang_x64/gen_snapshot
src % file out/linux_profile_arm/clang_x64/gen_snapshot 
out/linux_profile_arm/clang_x64/gen_snapshot: Mach-O 64-bit executable x86_64
swift-kim commented 2 years ago

Do you need to declare an additional parameter in gn? Anyway please consider contributing to https://github.com/flutter/buildroot.

WonyoungChoi commented 2 years ago

The --macos-build-tizen parameter can be added to https://github.com/flutter-tizen/engine/blob/flutter-2.8.1-tizen/tools/gn. And.... the modification of BUILDCONFIG is a little tricky, so it would be better to apply it as a patch in out CI build process.