[flutter_tools] Refactor Flutter SDK to support installation in read-only directory

[christopherfujino]

This would support being installed by root by a package manager. This would require:

• Providing an upgrade mechanism that is not dependent on git
• Allow the tool to determine its version without calling git commands that may write to .git
• Provide Flutter SDK version resolution (e.g. flutter --version) not dependent on git tags
• Moving the Flutter cache directory outside of the repo
• Tracking the Flutter SDK version file outside the repo (perhaps in the cache?)
• Ensuring that pub does not try to write to the repo while doing pub get

[NANASHI0X74]

This is relevant for nixpkgs too and we have some patches there to move the cache: https://github.com/NixOS/nixpkgs/pull/173200#issuecomment-1146783427 https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/flutter/patches/move-cache.patch Is this a reasonable way to do this?

[christopherfujino]

This is relevant for nixpkgs too and we have some patches there to move the cache: NixOS/nixpkgs#173200 (comment) https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/flutter/patches/move-cache.patch Is this a reasonable way to do this?

@NANASHI0X74 Oh neat! I would say the fact that the path is hard-coded is a bug, we would ideally have some method that can easily be configured, and all other code in the tool calls that method to determine the path to the cache.

[carterbox]

• Providing an upgrade mechanism that is not dependent on git
• Allow the tool to determine its version without calling git commands that may write to .git
• Provide Flutter SDK version resolution (e.g. flutter --version) not dependent on git tags
• Tracking the Flutter SDK version file outside the repo (perhaps in the cache?)

These all seem like functions that could be provided via Dart if Flutter was a Dart package.

[christopherfujino]

• Providing an upgrade mechanism that is not dependent on git

What exactly would the upgrade mechanism be? Especially if you're using the distro vendored version of the Dart SDK?

[carterbox]

Using the dart pub tool. If your Dart SDK is too old for the newest version of Flutter, then you upgrade it using the same method that used to install it.

Besides the convenience of quickly getting the latest Flutter release, what are the reasons for the existing upgrade mechanism? There must be some right? Otherwise, I don't see why the Flutter team would expend engineering effort to duplicate functionality already provided by the Dart package manager.

[carterbox]

Is it because the Flutter engine isn't actually a Dart package and can't be distributed vida dart pub?

[christopherfujino]

Using the dart pub tool. If your Dart SDK is too old for the newest version of Flutter, then you upgrade it using the same method that used to install it.

You would always have to update your Dart SDK. Thus, when users want a new version of Flutter, there would be a level of indirection, where they would need to update to the corresponding Dart SDK version that their desired Flutter version depends on, before they can get a valid pub version solve. I think this is counter-intuitive to how users reason about pub.

Also, if you're installing the Dart SDK distributed with your linux distro, you may not have options about which Dart version to install.

Besides the convenience of quickly getting the latest Flutter release, what are the reasons for the existing upgrade mechanism? There must be some right? Otherwise, I don't see why the Flutter team would expend engineering effort to duplicate functionality already provided by the Dart package manager.

The release infrastructure is not really duplicating what pub does. The paradigm for Flutter's release tooling is that the user selects a channel they are interested in (stable, beta, or master). By default (either with a git clone or via pre-built bundle on the website) they will receive the latest version of that release. They can continue to use a particular Flutter version for as long as it still builds, but can also easily upgrade to newer versions of that channel via the flutter upgrade command.

Pub on the other hand, is solving the problem where the user has a set of dependencies and constraints, and it does a version solve for determining the transitive set of all of dependencies and the "best" version that meets all constraints...and then it fetches them. From my understanding, vendoring the Flutter SDK via pub is only leveraging the last part, downloading a remote file, which the user could do with curl.

[carterbox]

The release infrastructure is not really duplicating what pub does.

Thanks for pointing this out. I assumed that one could easily subscribe to prereleases of a dart package, but it seems that Dart's version specification is not very complex. It only has like 4 comparison operators and no wildcards, so you can't ask for uuid:4*-beta

I think my whole argument is moot however because the Flutter engine isn't a Dart package and can't be distributed via Dart pub. Only the Flutter tools would be distributable. This wasn't immediately obvious to me because the Engine is its own repo.

[carterbox]

Let reconsider the checklist for a hypothetical read-only Flutter/Engine/Dart SDK installed using a system package manager.

• Providing an upgrade mechanism that is not dependent on git

The upgrade mechanism for read-only install is getting a new pre-build Flutter SDK from the distro/package manager. Read-only flutter does not update itself; it's read-only.

• Allow the tool to determine its version without calling git commands that may write to .git

git describe --tags doesn't write to disk

• Provide Flutter SDK version resolution (e.g. flutter --version) not dependent on git tags

Why? As mentioned above git describe is a read-only operation.

• Moving the Flutter cache directory outside of the repo

Which cache? The Dart .pub-cache or ./bin/cache? .pub-cache can already be redirected with the PUB_CACHE environment variable.

• Tracking the Flutter SDK version file outside the repo (perhaps in the cache?)

For read-only the version doesn't change, so it should be fine to keep inside the repo.

• Ensuring that pub does not try to write to the repo while doing pub get

Once again, if we're talking about the Dart .pub-cache we can just move that to a writable user directory using PUB_CACHE. For read-only Flutter all the lockfiles should just be in place for the repo because it's read-only.

[christopherfujino]

• Providing an upgrade mechanism that is not dependent on git

The upgrade mechanism for read-only install is getting a new pre-build Flutter SDK from the distro/package manager. Read-only flutter does not update itself; it's read-only.

You have to also consider users who are not on Linux, and may not have a package manager. If we force Windows users to use a package manager like chocolatey, the flutter team has to help maintain that workflow. I know with homebrew on macOS, we have resisted officially endorsing it because they do not provide a way for a team to own a package, and thus ensure that users who type brew install flutter will certainly get a build that was signed on Flutter's CI systems, rather than a malicious copy.

• Allow the tool to determine its version without calling git commands that may write to .git

git describe --tags doesn't write to disk

git fetch --tags does. Also, by default, it will sometimes try to either garbage collect or re-compress its data when you run what would appear to be read-only commands. I don't remember what errors off-hand, but at one point I installed flutter with root-only-write permissions and tested it out to create this list.

• Provide Flutter SDK version resolution (e.g. flutter --version) not dependent on git tags

Why? As mentioned above git describe is a read-only operation.

Ditto.

• Moving the Flutter cache directory outside of the repo

Which cache? The Dart .pub-cache or ./bin/cache? .pub-cache can already be redirected with the PUB_CACHE environment variable.

bin/cache.

• Tracking the Flutter SDK version file outside the repo (perhaps in the cache?)

For read-only the version doesn't change, so it should be fine to keep inside the repo.

This is true, but the mechanism for writing this is the tool. This should be an easy fix, though.

• Ensuring that pub does not try to write to the repo while doing pub get

Once again, if we're talking about the Dart .pub-cache we can just move that to a writable user directory using PUB_CACHE. For read-only Flutter all the lockfiles should just be in place for the repo because it's read-only.

We always vendor a pre-populated .pub-cache directory with pre-built SDK packages, so that users can immediately start development without network access. On the first run, if the user already has a user-global .pub-cache in their home directory, we will copy it there. See https://github.com/flutter/flutter/issues/53833 for more context. This can be worked around, but it would take some design considerations.

[carterbox]

• Providing an upgrade mechanism that is not dependent on git

The upgrade mechanism for read-only install is getting a new pre-build Flutter SDK from the distro/package manager. Read-only flutter does not update itself; it's read-only.

You have to also consider users who are not on Linux, and may not have a package manager. If we force Windows users to use a package manager like chocolatey, the flutter team has to help maintain that workflow. I know with homebrew on macOS, we have resisted officially endorsing it because they do not provide a way for a team to own a package, and thus ensure that users who type brew install flutter will certainly get a build that was signed on Flutter's CI systems, rather than a malicious copy.

For clarity, supporting read-only Flutter SDK does not preclude writable Flutter SDK. I agree that you should not force users to use a package manager. We're talking about modifying the SDK, so that it can function as a static system provided toolkit (to be provided by a distro or a system maintainer (person)).

We always vendor a pre-populated .pub-cache directory with pre-built SDK packages, so that users can immediately start development without network access.

This is not true for 3.3.10? The releases are pre-populated, but many of the flutter commands (including build) will fail without internet access (by calling pub get)! Are you referring to features not yet in a stable release?

[christopherfujino]

This is not true for 3.3.10? The releases are pre-populated, but many of the flutter commands (including build) will fail without internet access (by calling pub get)! Are you referring to features not yet in a stable release?

I did not say that ALL commands will work offline (notably if there is some dependency required remotely). Also, I think you need to provide the --offline flag in order for pub get to be invoked without trying to call the network.

[carterbox]

Hmmm... My mistake. I thought flutter build automatically calls pub get every time, but that seems to not be the case.

OK. Here's my plan of attack:

• [ ] Adjust flutter initialization/doctor scripts so they don't try to re-write ./bin/cache/*.stamp files if they already match the what is cached. libimodiledevice.stamp and usbmuxd.stamp seem to be updated when the artifacts don't change. Needs more investigation.

• [x] Do something with ./bin/cache/lockfile handling. So that if it is read-only, Flutter still checks the cache, but doesn't try to edit it. Can someone help with an explanation of what this file is locking? (Can overide with environment FLUTTER_ALREADY_LOCKED=true)

I think once these are implemented, it should be possible for a Flutter SDK to be initialized, and other non-priviledged users to use it as read-only.

As a reminder, I'm not worried about edits to .pub-cache because it can be moved using environment variables. I'm also not worried about the channel, upgrade, downgrade, or precache commands failing because if you're using a read-only SDK, those are not allowed. All of the other commands should work with read-only.

[carterbox]

Here's my proposal for working with read-only Flutter:

Modify Flutter so it only updates ./version if it is different. Add new --read-only option to CLI which hides commands that could update the Flutter SDK cache

Then,

1. Packager prepopulates the Flutter SDK cache with flutter precache -a
2. Packager marks the Flutter root as read-only
3. Package provides an alias flutter = FLUTTER_ALREADY_LOCKED=true flutter --read-only

Assuming the precache step cached all of the things properly, then the user should be able to run flutter without any blocking errors.

[carterbox]

I've implemented a prototype of the above comment available on my fork, but I've run into a snag. It seems that if you move PUB_CACHE (or create a new one), the Flutter CLI wants to rebuild the flutter_tools package. However, this involve writing to FLUTTER_ROOT/packages/flutter_tools (the pubspec.lock file is updated). Moving the PUB_CACHE should be expected for read-only installs because each user of the SDK should have their own PUB_CACHE.

I feel like there should be a way to compile these Dart packages once during setup, then reuse the binary instead of recompiling every time the Dart cache moves, but I'm not sure how that works.

[christopherfujino]

This is the latest NixOS patches that they maintain on the Flutter stable channel to get this working: https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/compilers/flutter/patches/flutter3

[christopherfujino]

Here's my proposal for working with read-only Flutter:

Modify Flutter so it only updates ./version if it is different. Add new --read-only option to CLI which hides commands that could update the Flutter SDK cache

Unfortunately today any Flutter command might hit the cache. I think the better solution would be to allow the cache to be user configurable, and instruct the user to set it to a user-owned path.

Assuming the precache step cached all of the things properly, then the user should be able to run flutter without any blocking errors.

I think there might be other things we write to the cache other than just caching binaries.

Also you have to consider the .git directory.

[Hari-07]

Providing an upgrade mechanism that is not dependent on git Allow the tool to determine its version without calling git commands that may write to .git Provide Flutter SDK version resolution (e.g. flutter --version) not dependent on git tags

For these things I think rather than hiding them (which may lead to confusion if some installations just straight up don't have some CLI commands for people trying to follow along online resources), each specific distribution should implement its own functionality as to what happens. It can even be as simple as just printing a message to the console.

For example if using a package manager, it'd obviously be preferred that the package manager be used to upgrade flutter , so if someone tries to do flutter upgrade, we can say

Since installed via , use package_manager> upgrade flutter

For flutter --version too we could do something similar, but also might be able to actually implement something that interfaces with the package manager's CLI to find the version

flutter channel I think can just be a print prompting people to install via the official archives, since I think package_managers (atleast brew) doesn't prefer putting multiple distribution channels of a package

Tracking the Flutter SDK version file outside the repo (perhaps in the cache?)

Should be kept track of using the package-manager. If our aim is to introduce a --no-version-check option even without a package manager to override this, then I think it makes sense to have a file in $FLUTTER_ROOT which can have data such as version (or maybe even with package managers we can have such a file, and this file can have info about which channel this flutter was installed through)

Unfortunately today any Flutter command might hit the cache. I think the better solution would be to allow the cache to be user configurable, and instruct the user to set it to a user-owned path.

While the user configurable cache path is a nice additional feature, is that required for making this read only, since if we move bin/cache into .pub_cache, (which according to #53833 is in the users home directory and thus editable) would that be an issue? Or is the intention to make a version of flutter which doesn't write anything to disk at all (which seems crazy given we wont even be able to add a package)

Ensuring that pub does not try to write to the repo while doing pub get

Again it should be writing to the home directory pub cache

[Hixie]

I'm not sure we should do this.

The Flutter SDK is designed specifically to allow people to step through the framework code, adding print and changing the code in other ways, to aid debugging and so forth. The correct way to install Flutter is in a writable user directory, where it can be manipulated via git.

I think it's fine for OS package managers to download Flutter, install it in userspace, and configure the user's path. But I don't think we should make the flutter tool work differently when installed via an OS package manager than when installed directly. That would complicate testing and support in a way that's quite expensive, it would confuse users with documentation that doesn't match what they find works on their machine, it would prevent them from using some of the debugging tools, etc.

[christopherfujino]

I'm not sure we should do this.

The Flutter SDK is designed specifically to allow people to step through the framework code, adding print and changing the code in other ways, to aid debugging and so forth. The correct way to install Flutter is in a writable user directory, where it can be manipulated via git.

This is a good point. In my opinion, the biggest reason we SHOULDN'T do this is developers would essentially be "stuck" using the version of the Flutter SDK that their package manager is packaging, unless for example Debian decided to package every stable Flutter as a different package. Even then, if there were a critical bug that was fixed in beta but not stable, they would have to go and clone flutter anyway.

[hacker1024]

This is a good point. In my opinion, the biggest reason we SHOULDN'T do this is developers would essentially be "stuck" using the version of the Flutter SDK that their package manager is packaging, unless for example Debian decided to package every stable Flutter as a different package. Even then, if there were a critical bug that was fixed in beta but not stable, they would have to go and clone flutter anyway.

While this is problematic for most package managers, Nix (and to an extent, Arch Linux's pacman) make it quite easy to rebuild a package with forked source code or additional patches applied.

This is part of the reason we'd like Flutter to work better with package managers - you get all the package manager's powerful features. Nix, for example, would let you apply different custom patches to Flutter per-project, which is a lot nicer than managing a clone manually.

The Flutter SDK is designed specifically to allow people to step through the framework code, adding print and changing the code in other ways, to aid debugging and so forth. The correct way to install Flutter is in a writable user directory, where it can be manipulated via git.

Flutter installed through a package manager does not necessarily have to be read only - temporary changes for debugging can be made, so long as users don't expect them to persist with upgrades. We'd like the Flutter tooling itself to avoid modifying its own installation directory as much as possible to make upgrades from a package manager easier.

[Hari-07]

rebuild a package

Since flutter doesn't really build from code as such, and just uses that dart code directly how exactly does this system work?

[jgfoster]

the biggest reason we SHOULDN'T do this is developers would essentially be "stuck" using the version of the Flutter SDK [initially installed].

In a classroom setting I would be happy to live with this limitation for the length of a term.

[Hixie]

The Flutter SDK is designed specifically to allow people to step through the framework code, adding print and changing the code in other ways, to aid debugging and so forth. The correct way to install Flutter is in a writable user directory, where it can be manipulated via git.

Flutter installed through a package manager does not necessarily have to be read only - temporary changes for debugging can be made, so long as users don't expect them to persist with upgrades. We'd like the Flutter tooling itself to avoid modifying its own installation directory as much as possible to make upgrades from a package manager easier.

Maybe what we should do is provide tooling that package managers can ship as "flutter" that is actually just a shell script that downloads flutter to the user's home directory. That way Flutter keeps working as designed, but you can obtain it via the package manager UI. Basically a /bin/flutter that does something like:

• if $HOME/flutter does not exist, git clone flutter to $HOME/flutter.
• Defer to $HOME/flutter/bin/flutter.