search

flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
164.51k stars 27.13k forks source link

Slow initial build time due to "skipping target" inefficient hashFiles computation. #55157

Open malkia opened 4 years ago

malkia commented 4 years ago

Steps to Reproduce

  1. flutter create p1
  2. cd p1
  3. flutter run -d linux OR flutter run -d windows OR (I guess mac, anything else)

I won't include logs, just the first lines of "flutter doctor -v"

[✓] Flutter (Channel master, v1.18.0-6.0.pre.82, on Linux, locale en_US.UTF-8) • Flutter version 1.18.0-6.0.pre.82 at /home/malkia/p/flutter/flutter • Framework revision f35b673f2b (20 hours ago), 2020-04-19 02:45:01 +0530 • Engine revision a5e0b2f2f2 • Dart version 2.9.0 (build 2.9.0-1.0.dev 5b19445d9c)

I've dig a little, and found out that calling, "flutter run -d linux" (or possibly other platforms) maybe slow due to the "skipping target" step: https://github.com/flutter/flutter/blob/cb8bafb38df02f63edaea76e0dc7a74380f47ce8/packages/flutter_tools/lib/src/build_system/build_system.dart#L684 calling computeChanges() here: https://github.com/flutter/flutter/blob/cb8bafb38df02f63edaea76e0dc7a74380f47ce8/packages/flutter_tools/lib/src/build_system/build_system.dart#L907 calling later hashFIles() https://github.com/flutter/flutter/blob/cb8bafb38df02f63edaea76e0dc7a74380f47ce8/packages/flutter_tools/lib/src/build_system/build_system.dart#L981

calling _hashFile(), which (function just down below):

final Digest digest = md5.convert(await file.readAsBytes());

I believe this is the cause of slowdown, and have put some more findings here: https://www.reddit.com/r/FlutterDev/comments/g31c2e/what_are_the_common_problems_you_are_facing_while/fnw96ku/?context=3

jonahwilliams commented 4 years ago

Reading through the thread, there are a few concerns that need to be balanced.

Relying on external programs to do hash computation means that we either need a fallback in case they are missing or increase the installation cost for users. In the past we've had issues with users installing new versions of common executables on linux platforms, like zip, which had slightly different functionality. This would cause things to blow up on their computers.

Similar to the above would be using dart:ffi functionality to call into native crypto packages. This could break in ways we're not familiar with.

another possibilities would be to spin up multiple isolates at the start of an assemble run, and then farm the hash computations out through those. That might help when the number of files is large, but will also slow down cases that hashed fewer files (because isolate spawning is slow).

Finally, someone could take a look at the hashing code we're using and probably optimize it a bit more. Right now it is reading from files in fairly small chunks and allocating the entire time. It could be restructured to re-use a single buffer and read large chunks.

jonahwilliams commented 4 years ago

You might also see if https://github.com/flutter/flutter/pull/53848 improves the speed enough that switching strategies isn't noticeable

jonahwilliams commented 4 years ago

I tried my hand at a hybrid approach:

  1. use dart for small files (less than X bytes)
  2. shell out to external program (if supported) for greater than X bytes

The thoughts are that we get some additionally concurrency from the external program for large files, while avoiding the overhead when the dart hashing would be fast enough. Also avoid isolates and dart:ffi entirely. Assumes that reading the file size doesn't add measurable overhead.

Here are some numbers building the macOS desktop (after #53848 landed) where I tweaked the number of X bytes

X = 100,000 Bytes

[   +3 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[   +9 ms] [   +9 ms] "flutter assemble" took 2,520ms.
jonahwilliams-macbookpro2:flutter_gallery jonahwilliams$ flutter build macos --release -v | grep assemble
[   +3 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[   +9 ms] [   +9 ms] "flutter assemble" took 2,459ms.

No change

[   +2 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[   +9 ms] [   +9 ms] "flutter assemble" took 2,517ms.
jonahwilliams-macbookpro2:flutter_gallery jonahwilliams$ flutter build macos --release -v | grep assemble
[   +2 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[   +9 ms] [   +9 ms] "flutter assemble" took 2,528ms.

X = 1 MB

[   +2 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[  +10 ms] [  +10 ms] "flutter assemble" took 1,406ms.
jonahwilliams-macbookpro2:flutter_gallery jonahwilliams$ rm ../../../bin/cache/flutter_tools.s*

jonahwilliams-macbookpro2:flutter_gallery jonahwilliams$ 
jonahwilliams-macbookpro2:flutter_gallery jonahwilliams$ flutter build macos --release -v | grep assemble
[   +2 ms] ♦ /Users/jonahwilliams/Documents/flutter/bin/flutter --verbose assemble -dTargetPlatform=darwin-x64 -dTargetFile=lib/main.dart -dBuildMode=release -dTreeShakeIcons=false -dDartObfuscation=false -dSplitDebugInfo= --DartDefines= -dExtraFrontEndOptions= --build-inputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterInputs.xcfilelist --build-outputs=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral/FlutterOutputs.xcfilelist --output=/Users/jonahwilliams/Documents/flutter/dev/integration_tests/flutter_gallery/macos/Flutter/ephemeral release_macos_bundle_flutter_assets
[  +10 ms] [  +10 ms] "flutter assemble" took 1,304ms.

This seems to be an improvement, provided we only use it on the large files

malkia commented 4 years ago

I believe the issue was due to lots of small files (633) in my case that had to be checked for md5 sums. e.g. after say "flutter linux -d linux" you can find them locally, these are just json files with "inputs:" and then lots of files listed. These files are checked for md5, and this is where the slowdown is coming (for me)


./.dart_tool/flutter_build/48b1e69a13fea91d19ffb9e8ed6035c8/unpack_linux_debug.stamp
./.dart_tool/flutter_build/48b1e69a13fea91d19ffb9e8ed6035c8/kernel_snapshot.stamp
./.dart_tool/flutter_build/48b1e69a13fea91d19ffb9e8ed6035c8/debug_bundle_linux_assets.stamp```

for example the first 10 from the "unpack_linux_debug.stamp"

```{"inputs":["/home/malkia/p/flutter/flutter/packages/flutter_tools/lib/src/build_system/targets/linux.dart"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/libflutter_linux_glfw.so"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/flutter_export.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/flutter_messenger.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/flutter_plugin_registrar.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/flutter_glfw.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/icudtl.dat"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/cpp_client_wrapper_glfw/include/flutter/json_method_codec.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/cpp_client_wrapper_glfw/include/flutter/json_message_codec.h"
"/home/malkia/p/flutter/flutter/bin/cache/artifacts/engine/linux-x64/cpp_client_wrapper_glfw/include/flutter/json_type.h"````
jonahwilliams commented 4 years ago

In my benchmarks, the small files take almost no time. Additionally we no longer perform hashing on anything in bin/cache