search

dart-lang / native

Dart packages related to FFI and native assets bundling.
BSD 3-Clause "New" or "Revised" License
155 stars 43 forks source link

[native_assets_cli] In-hook caching #1375

Open dcharkes opened 3 months ago

dcharkes commented 3 months ago

TODO

==================

How does a hook writer producing many assets, reuse their previous build in a next run? E.g. How do they cache assets?

For example: If we have 1000 3d models in source files, and they need to be compiled to some binary format, and we change one. Then the next cold start of the flutter app will rerun the hook, and if the hook writer does not get access to their previously built binary files, they will need to rerun the "compilation" step for all 1000 assets.

This is a simpler variant than thinking about caching with hot restart in mind, but both will need similar infrastructure.

We have two possible approaches here: (1) hook writers can check the timestamps of their own dependencies and the build outputs and on rebuild the ones which need rebuilding, or (2) the protocol reports changed dependencies and asset IDs that need rebuilding.

1. Hooks do caching internally

If the native assets builder promises to not modify the output directory in between runs, then hook writers can internally.

The hooks always output the full list of assets, and all the asset files exist after a hook run.

import 'dart:io';

import 'package:native_assets_cli/native_assets_cli.dart';

void main(List<String> args) async {
  await build(args, (config, output) async {
    final packageName = config.packageName;
    final sourceDirectory =
        Directory.fromUri(config.packageRoot.resolve('models/'));
    // If assets are added, rerun hook.
    output.addDependency(sourceDirectory.uri);

    await for (final sourceFile in assetDirectory.list()) {
      if (sourceFile is! File) {
        continue;
      }

      final targetFile = File.fromUri(output.outputDirectory.resolve(sourceFile.replaceLast('.model', '.binary'));
      if (!await targetFile.exists() || await sourceFile.lastModified() > await targetFile.lastModified()){
        compile(sourceFile, targetFile)
      }

      // The file path relative to the package root, with forward slashes.
      final name = targetFile
          .toFilePath(windows: false)
          .substring(config.packageRoot.toFilePath(windows: false).length);

      output.addAsset(
        DataAsset(
          package: packageName,
          name: name,
          file: targetFile.uri,
        ),
        linkInPackage: config.linkingEnabled ? packageName : null,
        dependencies: [dataAsset.uri],
      );
      output.addDependency(
        sourceFile,
      );
    }
  });
}

For example wrapping a CMake build already works this way by default, as CMake caches stuff internally in its build folder.

On thing to consider is if a hook has multiple builders adding assets. Then those builders should likely have their own subdirectory in the output directory.

TODO:

2. BuildConfig reports changed dependencies and assets which need to be rebuilt

An alternative design is to have hooks get a list of changed global dependencies and a map with changed asset ids as keys and which of the dependencies for each asset changed as values in the map.

Then the hook would only run the build for the changed assets and only report the changed assets.

An open question is whether this should be done inside the same directory as the initial build or not. If no, then no intermediate build artifacts can be used (CMake). If yes, we have to worry about combining all the build outputs from various runs.

It's not clear that this is beneficial for hook writers over using dart:io. But it does add a lot of extra book keeping to the native_assets_builder to combine the build outputs of various subsequent runs.

Hence, we suggest going with option 1.

Thanks @HosseinYousefi for triggering this discussion!

Background, only accessible for Googlers: http://go/dart-native-asset-caching

Somewhat related:

HosseinYousefi commented 3 months ago

If we're not doing 2, then "dependency" is simply thing that if changed, will trigger a build hook run on cold/hot restart. However if the user really wants to exploit this system, they can split up their package into several packages, each with a different build hook and a different set of dependencies, so that they can get a more fine grained rebuild.

dcharkes commented 1 month ago

We probably want to align in-hook caching with the hook caching:

If we end up using file-hashes for hook caching, we should do the same for inside the hooks.