`Bug/regression` | Flutter SDK can't be found in Flutter 3.22

Andreas02-dev commented 3 months ago

Describe the bug

The Flutter SDK cannot be found by build_runner and potentially other tools in the latest flutter version in nixpkgs (currently 3.22.0). In the previous nixpkgs Flutter version, which can now be found under flutter319, this worked correctly.

Steps To Reproduce

Steps to reproduce the behavior:

Create a Flutter project and add a library that utilizes code-generation with build_runner, such as Riverpod
Run the command dart run build_runner doctor -v (or any other command of build_runner, such as watch or build).

Observe the failure with the following output:

Resolving dependencies in `/home/andreas/Downloads/nixpkgs_flutter_repro/failing/test`... 
Because test depends on flutter_test from sdk which doesn't exist (the Flutter SDK is not
available), version solving failed.

I have also created a reproduction repository at https://github.com/Andreas02-dev/nixpkgs_flutter322_regression which also contains the reproduction steps for a working and failing example, where the only difference is using Flutter 3.19 (working) or Flutter 3.22 (failing).

Expected behavior

build_runner does not fail:

┌─ nixpkgs_flutter_repro/working/test is  v1.0.0+1 via  v3.3.2 via  impure (nix-shell-env) 
└─> dart run build_runner doctor -v
Building package executable... (4.1s)
Built build_runner:build_runner.
[INFO] Entrypoint:Generating build script...
[INFO] Entrypoint:Generating build script completed, took 192ms

[INFO] Bootstrap:Precompiling build script......
[INFO] Bootstrap:Precompiling build script... completed, took 2.7s

[FINE] Bootstrap:Core package locations file does not exist
[INFO] doctor:No problems found!

…nixpkgs_flutter_repro/working/test is  v1.0.0+1 via  v3.3.2 via  impure (nix-shell-env) took 7s 
└─> flutter --version
Flutter 3.19.4 • channel stable • https://github.com/flutter/flutter.git
Framework • revision nixpkgs000 () • 1970-01-01 00:00:00
Engine • revision a5c24f538d
Tools • Dart 3.3.2 • DevTools 2.31.1

Actual behavior

build_runner fails with the following output:

┌─ nixpkgs_flutter_repro/failing/test is  v1.0.0+1 via  v3.4.0 via  impure (nix-shell-env) 
└─> dart run build_runner doctor -v
Resolving dependencies in `/home/andreas/Downloads/nixpkgs_flutter_repro/failing/test`... 
Because test depends on flutter_test from sdk which doesn't exist (the Flutter SDK is not
  available), version solving failed.

Flutter users should use `flutter pub` instead of `dart pub`.

┌─ nixpkgs_flutter_repro/failing/test is  v1.0.0+1 via  v3.4.0 via  impure (nix-shell-env) 
└─> flutter --version
Flutter 3.22.0 • channel stable • https://github.com/flutter/flutter.git
Framework • revision nixpkgs000 () • 1970-01-01 00:00:00
Engine • revision f6344b75dc
Tools • Dart 3.4.0 • DevTools 2.34.3

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Notify maintainers

@babariviere @ericdallo @mkg20001 @RossComputerGuy @FlafyDev @hacker1024

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.34, NixOS, 24.05 (Uakari), 24.05.20240619.dd457de`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `""`
 - nixpkgs: `/etc/nixPath`

Add a :+1: reaction to issues you find important.

Andreas02-dev commented 3 months ago

I believe I have found the issue. As @maximoffua pointed out in https://github.com/maximoffua/flutter.nix/issues/19#issuecomment-2194391641, flutter pub run ... did work.

This lead me to look further into the Flutter source code, where we see that flutter pub run uses PackagesForwardCommand under the hood https://github.com/flutter/flutter/blob/ef34436402beaab4822a7306d9ffca17ebbdd2e3/packages/flutter_tools/lib/src/commands/packages.dart#L40C19-L40C41, with the description of running the pub tool in a Flutter context (https://github.com/flutter/flutter/blob/ef34436402beaab4822a7306d9ffca17ebbdd2e3/packages/flutter_tools/lib/src/commands/packages.dart#L124). Drilling deeper down, we see that the runCommand method calls pub.interactively. Looking at the default implementation reveals that it utilizes _computePubCommand which uses Cache.flutterRoot (https://github.com/flutter/flutter/blob/ef34436402beaab4822a7306d9ffca17ebbdd2e3/packages/flutter_tools/lib/src/dart/pub.dart#L548).

Putting 2 and 2 together, we find a patch in @maximoffua 's repository (https://github.com/maximoffua/flutter.nix/blob/main/pkgs/flutter/patches/flutter-pub-dart-override.patch) that modifies this part of Flutter in order for the flutter pub command to get the correct (containing the Flutter SDK) Dart SDK without using FlutterRoot.

Now that we know what makes dart run or dart pub run different from flutter pub run, it's trivial to find a solution. For this, we look at https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/cache.dart, which shows us we can allow Cache.flutterRoot to find the correct Flutter root path by setting the appropriately named environment variable FLUTTER_ROOT to point to "${pkgs.flutter}".

After setting this (https://github.com/Andreas02-dev/nixpkgs_flutter322_regression/blob/807dc5919b897e2f060726ae03ce05a8e64bee4a/fix/flake.nix#L24), we can now use dart run build_runner doctor -v in Flutter 3.22.

Now that we know what's happening and how we could solve it, we should figure out the best way to solve this: do we document that setting FLUTTER_ROOT is required for the dart run command to work with packages requiring the Flutter SDK, or do we try to patch this in somehow? I'm curious on what @maximoffua and the nixpkgs maintainers @babariviere @ericdallo @mkg20001 @RossComputerGuy @FlafyDev @hacker1024 think. We can discuss this here, or create a thread. If we create a thread, I would appreciate it if I could be kept in the loop.

Looking forward to hearing from you, Andreas

maximoffua commented 3 months ago

Thank you, @Andreas02-dev for this great investigation!

I think, it is sane to add the following to the flutter derivation:

# ...
shellHook = ''
  export FLUTTER_ROOT="${pkgs.flutter}"
'';

Will this work? I have never tried shellHook yet, however I have seen that some packages set extra env vars when they are used in profile/shell. Another question is how to reference self in this hook...

rob-mur commented 1 month ago

Hi - I just ran into this and can confirm that the suggested fix above does indeed work. Please see a minimal dev shell below that works!

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs =
    inputs:
    inputs.flake-utils.lib.eachDefaultSystem (
      system:
      let

        pkgs = import inputs.nixpkgs {
          inherit system;
          config = {
            allowUnfree = true;
          };
        };

        devShells.default = pkgs.mkShell
          {
            buildInputs = with pkgs;
              [
                flutter
              ];

            shellHook = ''
              export CHROME_EXECUTABLE=/run/current-system/sw/bin/google-chrome-stable
              export PATH="$PATH":"$HOME/.pub-cache/bin"
              export FLUTTER_ROOT="${pkgs.flutter}"

              fish
            '';
          };

      in
      {

        inherit devShells;

      }
    );

}

NixOS / nixpkgs