search

shorebirdtech / shorebird

Code Push for Flutter and other tools for Flutter businesses.
https://shorebird.dev
Other
2.25k stars 133 forks source link

fix: validate integrity of aot_tools.dill asset on disk #1771

Closed eseidel closed 2 months ago

eseidel commented 7 months ago

From discord: https://discord.com/channels/1030243211995791380/1030243213002412164/1213550586134265886

⠋ Linking AOT files...[Process.run] /Users/james/.shorebird/bin/cache/flutter/5b9d29d67adb059103beefb65710ee3dabae2f85/bin/dart run /Users/james/.shorebird/bin/cache/artifacts/aot-tools/62fa4e8bb30f56eb658989d3e6a982cb787d79d6/aot-tools.dill link --base=/var/folders/tq/pd1qk95j1hs9gfr8yws2m47w0000gn/T/TOpybX/Products/Applications/Runner.app/Frameworks/App.framework/App --patch=/Users/james/development/project/sport_mobile/build/out.aot --analyze-snapshot=/Users/james/.shorebird/bin/cache/flutter/5b9d29d67adb059103beefb65710ee3dabae2f85/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --output=/Users/james/development/project/sport_mobile/build/out.vmcode (in /Users/james/development/project/sport_mobile/build)
Exited with code 253

stderr:
Can't load Kernel binary: Invalid kernel binary: Indicated size is invalid.


✗ Failed to link AOT files: Exception: Failed to link: Can't load Kernel binary: Invalid kernel binary: Indicated size is invalid.
 (0.1s)

Why is dart pulling from 5b9d29d67adb059103beefb65710ee3dabae2f85 but aot-tools pulling from 62fa4e8bb30f56eb658989d3e6a982cb787d79d6?

https://github.com/shorebirdtech/engine/commit/62fa4e8bb30f56eb658989d3e6a982cb787d79d6

That appears to be Dart 3.3.0, which is the same one used by Flutter 3.19.2.

/Users/james/.shorebird/bin/cache/flutter/5b9d29d67adb059103beefb65710ee3dabae2f85/bin/dart --version Dart SDK version: 3.3.0 (stable) (Tue Feb 13 10:25:19 2024 +0000) on "macos_arm64"

eseidel commented 6 months ago

I replied to the user just now: https://discord.com/channels/1030243211995791380/1030243213002412164/1223121989573546094

eseidel commented 5 months ago

I'm not sure we have enough information to go on here? I think we should probably close this until we find another case.

lrferreiro commented 4 months ago

Fail when run patch, my app is available only in TestFlight.

Captura de pantalla 2024-05-31 a las 3 06 05 a m Captura de pantalla 2024-05-31 a las 3 15 26 a m
eseidel commented 3 months ago

I think this happens when we find the wrong app.dill? Logic which I think @bryanoltman recently improved in https://github.com/shorebirdtech/shorebird/pull/2194.

bryanoltman commented 3 months ago

Yeah, I think this was usually caused by selecting an incorrect app.dill.

Eldar2021 commented 3 months ago

I used Shorebird 1.1.15 version. my app is available only in TestFlight.

macbook_pro% shorebird patch ios
✓ Fetching apps (0.5s)
✓ Fetching releases (0.3s)
Which release would you like to patch? 1.0.26+37
✓ Fetching xcarchive artifact (0.3s)
✓ Downloading xcarchive (84.2s)
✓ Building patch with Flutter 3.22.2 (72996fba5d) (242.5s)
✓ Verifying patch can be applied to release (2.1s)
✓ Extracting release artifact (0.7s)
✗ Failed to link AOT files: Exception: Failed to link: Can't load Kernel binary: Invalid kernel binary: Indicated size is invalid.
 (1.5s)

If you aren't sure why this command failed, re-run with the --verbose flag to see more information.

Same problem!

bryanoltman commented 3 months ago

There should be a line immediately below this pointing to a log file. Can you share that? That will help us debug the issue.

TryImpossible commented 3 months ago 
barry@192 shorebird_example % shorebird doctor
Shorebird 1.1.15 • git@github.com:shorebirdtech/shorebird.git
Flutter 3.22.2 • revision 72996fba5d330551d694c6e09189e215c09b1c5b
Engine • revision ef99b23fc158eca48dd9ecb76a819329985ad628

✓ Shorebird is up-to-date (4.4s)
✓ Flutter install is correct (0.5s)
  [!] Shorebird does not respect the FLUTTER_STORAGE_BASE_URL environment variable at this time
✓ AndroidManifest.xml files contain INTERNET permission (95ms)
✓ Has access to storage.googleapis.com (1.7s)
✓ shorebird.yaml found in pubspec.yaml assets (8ms)
1 issue detected.

@bryanoltman @eseidel @lrferreiro shorebird patch --platforms=ios --release-version=1.0.1+2 --no-codesign， same problem! patch.log 1720247079401_shorebird.log

@bryanoltman , can you help me fix it

eseidel commented 2 months ago

Looks like this is the aot-tools.dill being missing/invalid?

2024-07-06T14:26:46.631709 [Process.run] /Users/barry/.shorebird/bin/cache/flutter/72996fba5d330551d694c6e09189e215c09b1c5b/bin/dart run /Users/barry/.shorebird/bin/cache/artifacts/aot-tools/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill link --help
eseidel commented 2 months ago

That dill should be downloaded from: https://storage.googleapis.com/download.shorebird.dev/shorebird/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

If you wanted to you could download a second copy with:

cd /tmp
curl -O https://storage.googleapis.com/download.shorebird.dev/shorebird/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

And then compare:

diff /tmp/aot-tools.dill /Users/barry/.shorebird/bin/cache/artifacts/aot-tools/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

That would tell us if there is some sort of cache corruption.

Flattih commented 2 months ago

I got the release with Shorebird, published it in the app store, then deleted the build file and when I tried to patch it, I got the following error: Fetching apps (0.5s) ✓ Fetching releases (0.3s) Which release would you like to patch? 1.2.0+1 ✓ Fetching xcarchive artifact (0.3s) ✓ Downloading xcarchive (3.8s) ✓ Building patch with Flutter 3.22.2 (72996fba5d) (176.6s) ✓ Verifying patch can be applied to release (1.1s) ✓ Extracting release artifact (0.3s) ✗ Failed to link AOT files: Exception: Failed to link: Can't load Kernel binary: Invalid kernel binary: Indicated size is invalid.

TryImpossible commented 2 months ago

That dill should be downloaded from: https://storage.googleapis.com/download.shorebird.dev/shorebird/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

If you wanted to you could download a second copy with:

cd /tmp
curl -O https://storage.googleapis.com/download.shorebird.dev/shorebird/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

And then compare:

diff /tmp/aot-tools.dill /Users/barry/.shorebird/bin/cache/artifacts/aot-tools/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill

That would tell us if there is some sort of cache corruption.

barry@192 Desktop % diff aot-tools.dill /Users/barry/.shorebird/bin/cache/artifacts/aot-tools/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill 
Binary files aot-tools.dill and /Users/barry/.shorebird/bin/cache/artifacts/aot-tools/ef99b23fc158eca48dd9ecb76a819329985ad628/aot-tools.dill differ

I replaced the file aot tools. dill and the problem was resolved

Flattih commented 2 months ago

Thanks, this solved my problem too.

eseidel commented 2 months ago

Thanks. We'll fix this to validate this asset on disk (we already have support for bundletool.jar validation I believe).

eseidel commented 2 months ago

To fix this we should:

  1. Make sure we download to a temporary directory.
  2. Verify the integrity of the download by checking against the hash (optional).
  3. Move the zip (or directly unzip from the temp) into a new tmp.
  4. Clear any existing artifacts
  5. Move (the only atomic filesystem operation) the unzipped directory into the desired location.
  6. Presumably write a stamp file? (Not sure if they do this) to record success?

Doing basically whatever flutter does for their assets (which is probably what I describe above) will make all of these issues go away.