Automatic screenshot generation for Marketing & Tests

M123-dev commented 3 years ago

What

We have around 80 languages in which we have products.
We have around 5 screenshots to make
A screenshot might need up to 10 scanned products.
The time to produce local feeling screenshots that are up to date with the current UI is tremendous.
And yet: “a majority of potential users will not read through the whole app description to find out, what your app is about. 50% of users make their purchase decisions based on the first impression”

Tasks

### Write UI test to create the screenshots
  - [ ] #1447
  - [ ] #1449
  - [ ] #1446
  - [ ] #1448
  - [x] #1596
  - [ ] #1433

Other tasks

[ ] Automate adding frames and slogans around screenshots
- [ ] fastlane frameit
[ ] Set up translation of slogans with Crowdin
[x] #539

How

Design doc Since Fastlane has no direct support for flutter screenshots, I think this could help.
https://medium.com/@nocnoc/automated-screenshots-for-flutter-f78be70cd5fd
https://github.com/mmcc007/screenshots
https://docs.fastlane.tools/getting-started/ios/screenshots/

Part of

536

rolandgeider commented 3 years ago

Judging from the issues in the repo, it seems the package is not maintained anymore, which is a pity

M123-dev commented 3 years ago

@rolandgeider oh that's a shame, have you found an alternative for your project?

rolandgeider commented 3 years ago

sadly not, but I also haven't looked much into it. It's somehow possible to save the current screen contents to a file, but nothing so easy as that package, I was hoping you guys would have some ideas 😅

M123-dev commented 3 years ago

@rolandgeider I have not yet dealt with the topic much more either, if I find a solution I will let you know.

rolandgeider commented 3 years ago

FYI, I have found this which looks promising, using only the flutter driver: https://medium.com/flutter-community/testing-flutter-ui-with-flutter-driver-c1583681e337

however it seems flutter_driver and flutter_launcher_icons have dependency incompatibilities 🙄 so I couldn't test it

M123-dev commented 3 years ago

@rolandgeider I don't think we have a real alternative to the screenshot package.

Currently only a not merged fork supports null-safety. Its not future proof but its probably the best way to go for now.

M123-dev commented 3 years ago

Someone suggested to transfer the package to the FlutterCommunity, hopefully it gets accepted.

https://github.com/fluttercommunity/community/issues/74

monsieurtanuki commented 3 years ago

For the record, in another project I developed a shell that can change the language of the iOS simulator and run tests - which include screenshots. Basically, on a mac book and not 100% automatic but I just have to start the right simulator and run the shell: all the screenshots of all languages will be created for that simulator. Then do it again for the other 2 screen configurations/simulators needed in the AppStore.

M123-dev commented 3 years ago

Two days ago in the fellowship frontend workgroup sync @gspencergoog mentioned screenshot tests. That gave me the idea that we might be able to abuse a screenshot testing tool to generate screenshots for fastlane. I couldn't find the one he mentioned but golden_toolkit (by ebay) looks very promising.

A article with some additional information: https://medium.com/flutter-community/flutter-screenshot-testing-as-a-solid-ui-regression-tool-630221a621e4

gspencergoog commented 3 years ago

The one Flutter uses is Skia Gold. I don't think we can use their servers directly, unless we want to just use their code and set up our own instance that uses Google Cloud (it is open source, at least). I could ask the Skia team how hard/expensive it is to set up if you like.

monsieurtanuki commented 3 years ago

Anyway we need iOS screenshots (minimum is one in 3 different screen sizes in English), without that it's not possible to have it in the AppStore (maybe OK in TestFlight, though).

M123-dev commented 3 years ago

@gspencergoog thanks for the offer, but I think it is easier and more uniform to do the process (at least for generations not for tests) with the goolden_toolkit and github actions.

We only have to think about if plain screenshots are to booring and if so how we edit them.

monsieurtanuki commented 3 years ago

Hey @M123-dev, you're supposed to be on vacation! ;)

gspencergoog commented 3 years ago

@M123-dev That makes sense.

teolemon commented 2 years ago

@monsieurtanuki note: I've managed to automate running the Screenshot tests suites on GitHub Actions for the classic apps, so we can run them on all platforms without blocking a personal computer

monsieurtanuki commented 2 years ago

@teolemon That means classic Android and classic iOS apps, but not flutter so far, right?

teolemon commented 2 years ago

Yes

M123-dev commented 2 years ago

Just did some further research and found another approach that doesn't use the not null safe screenshots package

https://blog.codemagic.io/flutter-automated-screenshot-testing/

teolemon commented 2 years ago

Is this an issue we can split for GSOC, asking newcomers to write one script each ?

M123-dev commented 2 years ago

What do you mean with one script for each, one for iOS one for android?

I still think it's better to somehow do it in flutter but when there is no other way we could do it like that.

Another idea which just came to my mind. Running the app in debug mode in the browser, there we can switch the device and language and automate this with puppeteer. This probably won't run on github actions though

monsieurtanuki commented 2 years ago

@teolemon @M123-dev For my flutter/iOS app, months ago, I had to create a "test" that included screenshots (then I could use the same test for Android of course). And I had to run it with unix scripts again and again, for different languages (env variable). Can't remember if I could restart automagically the iOS simulator or if I had to do it manually. Can check that. Would that be compatible with github actions? Is fastlane planning to include iOS screenshots one day?

teolemon commented 2 years ago

One script per scenario I meant @M123-dev . Same tech, but there are many screenshots to take @monsieurtanuki We have already automated it for the classic apps, for both platforms (ios and android) using github actions, we should be able to scrap the .yml and the fastlane parts, the Flutter tests would need to be written

https://github.com/openfoodfacts/openfoodfacts-ios/blob/develop/.github/workflows/screenshots.yml https://github.com/openfoodfacts/openfoodfacts-androidapp/pull/4206/files

monsieurtanuki commented 2 years ago

I guess the first big step would be to create the first test w/ screenshot and check that we're able to use that test to generate screenshots in tons of languages. And that the test is easy to copy/paste. Then, perhaps split that test in several one-screenshot tests, though I think we could do everything in the same test session (at least that's what I did in my own project).

teolemon commented 2 years ago

@monsieurtanuki @M123-dev I've started the GitHub actions part in #1208

monsieurtanuki commented 2 years ago

@teolemon What do you need now? A test that takes one screenshot, and optionally a unix script that runs it?

teolemon commented 2 years ago

I'm not sure how the test should be written and how they should be run, but at the end of the day Github Actions can either run unix command, or run emulators (spawning an emulator is already part of my PR). So no strong opinions.

alphaNewrex commented 2 years ago

Two days ago in the fellowship frontend workgroup sync @gspencergoog mentioned screenshot tests. That gave me the idea that we might be able to abuse a screenshot testing tool to generate screenshots for fastlane. I couldn't find the one he mentioned but golden_toolkit (by ebay) looks very promising.

A article with some additional information: https://medium.com/flutter-community/flutter-screenshot-testing-as-a-solid-ui-regression-tool-630221a621e4

Has this package been tried and tested? Looks promising We can generate multiple Devices Screenshot along with Multiple parameterized screenshot for different languages

monsieurtanuki commented 2 years ago

@alphaNewrex I ran some tests with golden_tookit but they were a bit disappointing - good enough for tests, but not for proper screenshots: my_widget-example

monsieurtanuki commented 2 years ago

I've finally managed to create again screenshots on both Android and iOS. Done. Wasn't easy because of bugs and evolutions, but in the end it's not that hard.

The first (tricky) part is a fix of an flutter/iOS bug (cf. https://github.com/flutter/flutter/issues/91668):

go to flutter/packages/integration_test/ios/Classes/IntegrationTestPlugin.m
edit method registerWithRegistrar this way
(void)registerWithRegistrar:(NSObject
*)registrar { [[IntegrationTestPlugin instance] setupChannels:registrar.messenger]; }
then flutter clean

Then, let's create a new project (flutter create tanuki_project)

Edit pubspec.yaml:

dev_dependencies:
  integration_test: # add
    sdk: flutter    # add
  flutter_test:
    sdk: flutter

Create 2 folders at the same level as lib: test_driver and integration_test.

File test_driver/screenshot_driver.dart:

import 'dart:io';
import 'package:integration_test/integration_test_driver_extended.dart';

// cf. https://dev.to/mjablecnik/take-screenshot-during-flutter-integration-tests-435k
Future<void> main() async {
  try {
    await integrationDriver(
      onScreenshot: (String screenshotName, List<int> screenshotBytes) async {
        final File image = await File('screenshots/$screenshotName.png')
            .create(recursive: true);
        image.writeAsBytesSync(screenshotBytes);
        return true;
      },
    );
  } catch (e) {
    print('Error occured: $e');
  }
}

File integration_test/app_test.dart:

import 'dart:io';
import 'package:flutter/foundation.dart' show kIsWeb;
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

import 'package:tanuki_project/main.dart' as app;

Future<void> takeScreenshot(tester, binding, name) async {
  if ((!kIsWeb) && Platform.isAndroid) {
    await binding.convertFlutterSurfaceToImage();
    await tester.pumpAndSettle();
  }
  await binding.takeScreenshot(name);
}

void main() {
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('end-to-end test', () {
    testWidgets('tap on the floating action button, verify counter',
        (WidgetTester tester) async {
      app.main();
      await tester.pumpAndSettle();

      // Verify the counter starts at 0.
      expect(find.text('0'), findsOneWidget);

      await takeScreenshot(tester, binding, 'test-screenshot-1');

      // Finds the floating action button to tap on.
      final Finder fab = find.byTooltip('Increment');

      // Emulate a tap on the floating action button.
      await tester.tap(fab);

      // Trigger a frame.
      await tester.pumpAndSettle();

      // Verify the counter increments by 1.
      expect(find.text('1'), findsOneWidget);
    });
  });
}

And the actual test run:

flutter drive --driver=test_driver/screenshot_driver.dart --target=integration_test/app_test.dart

The result here will be a file called test-screenshot-1.png in (new) folder screenshots.

monsieurtanuki commented 2 years ago

As a test, I'm going to create screenshots of the first (onboarding) page. We'll build up from there.

monsieurtanuki commented 2 years ago

That was painful, but I did it (iPhone 8 plus Simulator)! test-screenshot-2

monsieurtanuki commented 2 years ago

@teolemon I need your help: the cameras don't work on the emulator, and I have to replace them withpngs. Would you provide one for the screenshots? We'll see later which actual sizes we need.

teolemon commented 2 years ago

monsieurtanuki commented 2 years ago

Thank you @teolemon! That said, as in smoothie there are many controls on the bottom part, we'd be better off with interesting data like barcodes on top of the image. Localizations may be an issue too. But that's a start.

monsieurtanuki commented 2 years ago

I could not test it before for Android, but here we are: test-screenshot-onboarding-home

For the record, that confirms my low esteem for Stack and Position widgets.

teolemon commented 2 years ago

auto 407 / 670, for the record

g123k commented 2 years ago

Mmm why I am assigned to this issue?

teolemon commented 2 years ago

unsure, fixing this

openfoodfacts / smooth-app