Tests failing on the other machines

[InAnadea]

I created a golden test on my local machine. But after I pushed changes to the server test checks failed. Also on some other computers test tests fail.

OS is macos.

local machine logs macos 12.6 (21G115)

✓ Authentication screen golden

ci machine logs macOS 11.7 20G817 (macos-latest in the workflow)

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ The following assertion was thrown while running async test code: Golden "goldens/authentication_screen_golden.png": Pixel test failed, 0.08% diff detected. Failure feedback can be found at /Users/runner/work/metrics/metrics/test/presentation/failures

When the exception was thrown, this was the stack:

0 LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:102:7)

(elided one frame from package:stack_trace) ════════════════════════════════════════════════════════════════════════════════════════════════════ 00:50 +51 -1: /Users/runner/work/metrics/metrics/test/presentation/auth_screen_test.dart: Authentication screen golden [E] Test failed. See exception logs above. The test description was: Authentication screen golden 00:50 +51 -1: Some tests failed. Error: Process completed with exit code 1.

flutter doctor

Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.3.2, on macOS 12.6 21G115 darwin-arm, locale en-GB) [✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 14.0) [✓] Chrome - develop for the web [✓] Android Studio (version 2020.3) [✓] IntelliJ IDEA Community Edition (version 2021.3) [✓] VS Code (version 1.71.2) [✓] Connected device (3 available) [✓] HTTP Host Availability

• No issues found!

[coreysprague]

@InAnadea do your goldens include text being rendered with real fonts?

If so, I suspect you may get differences due to the OS version difference. If you analyze the diff, I suspect it's around aliasing of the fonts.

[InAnadea]

Hi @coreysprague! Yes, goldens includes text. How can I fix it? The fonts are included in the assets and flutter_test_config.dart contains await loadAppFonts().

[tsimbalar]

I'm not 100% sure, but I think we've had similar issues in the past... roughly, the rendering of Flutter is not exactly the same on different versions of macOS ...

i.e. the golden files generated on your v12.6 are not the same that would be generated on the v11.7 of your CI pipelines ...

is there any chance you can upgrade your pipelines to use the same version of macos you use for development ?

I also recall about some particular setting related to font-rendering on macos (used to be in Preferences ... but then was removed and became a flag you set through the command-line)

[tsimbalar]

here is an issue from the pass related to similar font-rendering problems:

• https://github.com/flutter/flutter/issues/56383#issuecomment-630345985

[InAnadea]

I added some updates to the CI script. I'm using the same os version (macos-12 in the workflow) and the same version of flutter. Also, I removed font smoothing for the golden test and golden generate.

Here is the GitHub actions workflow:

name: PR

on:
  workflow_dispatch:
  pull_request:
    branches: [ master, develop ]

jobs:
  check:
    name: PR

    runs-on: macos-12

    steps:
      - uses: actions/checkout@v3

      - name: 👮 Setup fvm
        uses: kuhnroyal/flutter-fvm-config-action@v1

      - name: 🖥️ Setup Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: ${{ env.FLUTTER_VERSION }}
          channel: ${{ env.FLUTTER_CHANNEL }}

      - name: ⬇️ Install dependencies
        run: dart pub get

      - name: 🏗️ Flutter build_runner build
        run: flutter pub run build_runner build --delete-conflicting-outputs

      - name: 🛠 Disable font smoothing
        run: defaults -currentHost write -g AppleFontSmoothing -int 0

      - name: 🧪 Test
        run: flutter test

      - name: 🛠 Enable font smoothing
        run: defaults -currentHost write -g AppleFontSmoothing -int 3

And command for local goldens update:

defaults -currentHost write -g AppleFontSmoothing -int 0
fvm flutter test --update-goldens
defaults -currentHost write -g AppleFontSmoothing -int 3

After these updates, the tests passed!

[Kiruel]

Your solution not work 100% of the time @InAnadea , sometimes it pass, sometimes fail with the same workflow files.