Open ductranit opened 2 years ago
@ductranit
Since you are running the integration test on a device, can you try to bundle them in as an asset and then try to access it using rootBundle
?
@ductranit Since you are running the integration test on a device, can you try to bundle them in as an asset and then try to access it using
rootBundle
?
I didn't try it yet, but I need to access the system path to load custom fonts (for golden test), load some test configs, take some screenshots & save to files. How can bundle support for that? I noticed that my test code works when I run integration test on ios, but not for android.
How can bundle support for that?
Since you are saving to the files, and the same file if you add it to pubspec.yaml under assets
and refer it to something like await rootBundle.loadString('assets/my_text.txt');
@darshankawar So I tried to read by rootBundle
& it works
final bytes = await rootBundle.load('assets/data.json');
But as I said, the problem is that I need to write screenshot to file which bundle can't help.
I tried to save screenshot with the path like:
screen.png
/Users/admin/Documents/flutter/my_app/assets/screen.png
assets/screen.png
and none of them can work
await binding.convertFlutterSurfaceToImage();
await tester.pumpAndSettle();
final data = await binding.takeScreenshot(name);
final path = 'screen.png'; // or `/Users/admin/Documents/flutter/my_app/screen.png` or `assets/screen.png`
final imageFile = File(path);
await imageFile.create(recursive: true);
await imageFile.writeAsBytesSync(data);
It throws exception when write file:
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following FileSystemException was thrown running a test:
Cannot create file, path = 'screen.png' (OS Error: Read-only file system, errno = 30)
When the exception was thrown, this was the stack:
#0 _File.create.<anonymous closure> (dart:io/file_impl.dart:255:9)
<asynchronous suspension>
<asynchronous suspension>
(elided 5 frames from dart:async and package:stack_trace)
The test description was:
Test app flow
════════════════════════════════════════════════════════════════════════════════════════════════════
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following message was thrown:
Multiple exceptions (5) were detected during the running of the current test, and at least one was
unexpected.
P/s: I had to change FlutterFragmentActivity
to FlutterActivity
to take screenshot because of this issue, but it will have a trouble because other plugins require fragment activity
@ductranit
Thanks for the update. Since it's throwing The following FileSystemException was thrown running a test: Cannot create file, path = 'screen.png' (OS Error: Read-only file system, errno = 30)
, can you try to pass full path ? ie instead of relative path, try to pass absolute path.
As I said above, all kinds of path have exception.
This is the full path /Users/admin/Documents/flutter/my_app/assets/screen.png
when I create file
await imageFile.create();
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following FileSystemException was thrown running a test:
Cannot create file, path =
'/Users/admin/Documents/flutter/my_app/assets/screen.png' (OS Error: No such
file or directory, errno = 2)
When the exception was thrown, this was the stack:
#0 _File.create.<anonymous closure> (dart:io/file_impl.dart:255:9)
<asynchronous suspension>
<asynchronous suspension>
(elided 5 frames from dart:async and package:stack_trace)
The test description was:
Test app flow
════════════════════════════════════════════════════════════════════════════════════════════════════
07:12 +0 -1: Test app flow [E]
Test failed. See exception logs above.
The test description was: Test app flow
If I tried with await imageFile.create(recursive: true);
, then it throws this exception
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following FileSystemException was thrown running a test:
Creation failed, path = '/Users' (OS Error: Read-only file system, errno = 30)
When the exception was thrown, this was the stack:
#0 _Directory.create.<anonymous closure> (dart:io/directory_impl.dart:117:11)
<asynchronous suspension>
<asynchronous suspension>
(elided 5 frames from dart:async and package:stack_trace)
The test description was:
Test app flow
════════════════════════════════════════════════════════════════════════════════════════════════════
06:18 +0 -1: Test app flow [E]
Test failed. See exception logs above.
The test description was: Test app flow
The problem is that the root path of integration test is /
- this is the root dir on my mac device and it's readonly
Thanks for the update. I see what you are saying. Can you provide us a minimal code sample, ie, main.dart
and integration_test -> app_test.dart
file that we can directly use and verify it further ?
@darshankawar Here is it ui_test.zip
This sample is just created by flutter create
on latest master
I run the test by flutter test integration_test
Thanks for the info. I verified this on Android and get same behavior / exception on latest stable and master. Keeping it open for further insights from the team.
/cc @jiahaog
Are you running the iOS integration test on a simulator? I'm not sure if that will work on a physical device.
This is probably happening because of the different filesystems used by the device. On iOS simulators, the filesystem of the simulator is the same as the host. Similarly for unit tests, the test run within the flutter tester device which runs on the host and has access to the same host filesystem. This is not the case on iOS physical devices or Android emulators or Android physical devices, which do not share the filesystem of the host.
@jiahaog yes, I run it on ios simulator. I just try on ios device and got the same android issue.
But if the device file system is different with the host, then how can I write file into it during the test?
As you can see above, I was trying to use relative path like screen.png
or assets/screen.png
but none of them can work.
any update? same problem~
actually, it seems like golden test doesn't work in integration test
actually, it seems like golden test doesn't work in integration test
I stumbled across the same issue. Any solutions?
@lexxxel @00SunnyDay00 I have to put the golden images into flutter assets (define in pubspec.yaml), then copy those golden files into internal storage of simulator/device when starting app and compare goldens
@ductranit so you can't use flutter test integration_test --update-golden
to automatically create/update those screenshots AND you have to have a separate pubspec.yaml for integration tests (because nobody wants to deploy this screenshots to production)?
If so, that makes golden integration tests unpractical.
@lexxxel actually I did a trick for --update-golden
: I upload the new golden into our server, and I have a script to download latest golden before running integration tests.
And yes for a separate pubspec.yaml.
I run tests & deploy build from CI system, and this issue stops me to do normally.
I use separate script for deploy & test app to generate pubspec.yaml & goldens.
I know this is an ugly hack,but there is no better way unless flutter dev supports for it.
@ductranit I might found a 'better' solution. Requirement: a emulator with root access. I do use (minimal example):
Directory tempDir = await getTemporaryDirectory();
String tempPath = tempDir.path;
await expectLater(introText, matchesGoldenFile(tempPath + "/QrScannerPageModel.verify.png"));
to push the .png into a folder that is predictable and adb pull/push
within the ci pipeline to move the files onto the emulator
I had the same problem with this issue So when I run integration test, the
Directory.current
is/
- this is the root dir on my mac device and it's readonly. I was trying to read json file & custom font in the test, so I have to pass the real path on my mac like:If I run command
flutter test integration_test
on ios, it works fine, I can load custom font, json & can also take screenshot with this path. But when I run on android device, it throw file not found error with the same path when I load font, jsonSo my questions are:
/
as rootflutter doctor -v
``` [✓] Flutter (Channel master, 2.6.0-12.0.pre.826, on Mac OS X 10.15.7 19H1519 darwin-x64, locale en-US) • Flutter version 2.6.0-12.0.pre.826 at /Users/admin/Documents/Soft/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision 6c1455a23a (25 hours ago), 2021-11-23 21:13:05 -0500 • Engine revision 539239e256 • Dart version 2.16.0 (build 2.16.0-42.0.dev) • DevTools version 2.8.0 [✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0) • Android SDK at /Users/admin/Library/Android/sdk • Platform android-31, build-tools 31.0.0 • ANDROID_HOME = /Users/admin/Library/Android/sdk • ANDROID_SDK_ROOT = /Users/admin/Library/Android/sdk • Java binary at: /Applications/Android Studio 4.2 Preview.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165) • All Android licenses accepted. [!] Xcode - develop for iOS and macOS (Xcode 12.4) • Xcode at /Applications/Xcode.app/Contents/Developer ! Flutter recommends a minimum Xcode version of 13. Download the latest version or update via the Mac App Store. • CocoaPods version 1.11.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2020.3) • Android Studio at /Applications/Android Studio 4.2 Preview.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165) [✓] VS Code (version 1.58.2) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.24.0 [✓] VS Code (version 1.63.0-insider) • VS Code at /Applications/Visual Studio Code - Insiders.app/Contents • Flutter extension version 3.28.0 [✓] Connected device (2 available) • macOS (desktop) • macos • darwin-x64 • Mac OS X 10.15.7 19H1519 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 96.0.4664.55 ```