search

dart-lang / pub

The pub command line tool
https://dart.dev/tools/pub/cmd
BSD 3-Clause "New" or "Revised" License
1.04k stars 228 forks source link

offline archive #4353

Open jwinarske opened 2 months ago

jwinarske commented 2 months ago

I have a need to support Long Term Support (LTS); automotive and IOT.

I'm proposing a new sub command called archive.

This would cache source types hosted and git into an archive folder.

Then without Internet access you can restore said archive, and build.

I have an implementation in Python that does this today. I am using this with the Yocto build system: https://github.com/meta-flutter/meta-flutter/blob/scarthgap/tools/pubspec.py

sigurdm commented 2 months ago

We don't have a separate command for this today, but you can already do:

PUB_CACHE=.pub_cache dart pub get

Which would put the cache inside the current directory. This way you can zip everything up.

Then when extracting somewhere run

PUB_CACHE=.pub_cache dart pub get --offline

To fix the absolute references in the .dart_tool/package_config.json

jwinarske commented 2 months ago

The use case is building 100+ apps simultaneously with LTS support. The proposed archive method dramatically decreases build storage. As there is only one archive of a package, which may show up 100+ times. The compress pub cache approach uses much more disk space at scale. The pub cache restore step takes ~1 second when multi-threaded.

sigurdm commented 2 months ago

Not sure I follow the use case exactly:

You have 100 apps, and now you want to make a single archive with all of them, or one archive for each?

Can you spell out what exactly the proposed command would do?

jwinarske commented 2 months ago

@sigurdm Archive would not include any redundant packages; only single references for git and hosted artifacts.

  1. pub archive Referencing the pubspec.lock file fetch hosted tar.gz, create tar.gz.256, and create .cache folder with -advisories.json, and -version.json files. This represents the archive; all the items fetched from the pub server.

  2. pub archive restore Referencing the pubspec.lock creates a pub cache using archive path. Restores pub cache to path specified by PUB_CACHE.

sigurdm commented 2 months ago

Archive would not include any redundant packages;

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

only single references for git and hosted artifacts.

What are "references" in this context?

jwinarske commented 2 months ago

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

Yes this is true for a single application. If I look at all the artifacts fetched for 100 apps, there are many common packages shared between them. This means the archive folder would only contain one copy of each package version. Using a compressed pub archive duplicates the same package version many times; thus increasing disk space.

What are "references" in this context?

These would be considered a reference:

jwinarske commented 2 months ago

Archive listing for flutter-gallery archive_example.txt

File sizes

Flutter Gallery