search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.28k stars 1.58k forks source link

unbundling dependencies (distro packaging) #51535

Open selfisekai opened 1 year ago

selfisekai commented 1 year ago

port analogical solutions from chromium:

I'm working on the same issue in flutter engine as well (flutter/flutter#121106, flutter/buildroot#687) - should at least some of the replacement build files be in a repository shared with flutter? (some are specific to flutter engine, some are shared with dart)

dependency-independent:

dependency list:

sortie commented 1 year ago

Hi, I build the release engineering infrastructure for Dart and I'd love for Dart to better fit distros, I'd be interested in this work and potentially helping out. I'll look a bit more into your links. What sort of specific changes did you have in mind?

selfisekai commented 1 year ago

using shared C libraries provided by the system, for C dependencies that Dart uses, e.g. zlib, icu, tcmalloc. currently, gclient downloads source code for all of them together with the source code for Dart, and they get compiled in to the binaries that use them. the standard within distributions is that these dependencies are separately packaged in the distribution's repositories, and are made available to other packages that utilize them, e.g. by querying pkgconf/pkg-config.

chromium achieves this with a script that replaces the gn files in the tree (e.g. //third_party/icu/BUILD.gn is getting replaced with https://chromium.googlesource.com/chromium/src/build/+/refs/heads/main/linux/unbundle/icu.gn). distributions run that script in the build process (before gn/ninja) with a list of dependencies to be replaced.

sortie commented 1 year ago

This sounds doable yeah. We have a high level tools/build.py entry point that invokes gn so I'm sure it could have a new option that makes it unbundle the dependencies, perhaps by passing a parameter down through the gn logic. I think that's a cleaner solution than invoking a script to rewrite the tree :)

sortie commented 1 year ago

Do you want to give this work an initial try? I'm happy to do the code reviews to make this happen and help discuss how it can be done :)

Something like this is also on our roadmap to improve the build-from-source journey although it's a little down the road

selfisekai commented 1 year ago

what about flutter engine in this case? also the build workflows for dependencies are coming from other projects, as I see: zlib, icu and tcmalloc (edit: tcmalloc gn files are in the sdk repo) from chromium, protobuf from fuchsia

sortie commented 1 year ago

I'm not sure about Flutter. Their infrastructure is a bit ever-changing and we constantly roll Dart into it via some complicated multi-headed solutions. If we at least solve the problem for Dart, we can see what it would take to do the equivalent work for Flutter. We're happy to try to collaborate with our friends in Chromium and Fuchsia if we need to amend or fork shared code :)

selfisekai commented 1 year ago

@sortie submitted #51612 and #51614 so far.

I've got changes for icu done as well, but:

  1. use_system_icu=true without removing some files from //third_party/icu/ ends up with linker errors. chromium has a tool for removing those - should I do the same here?
  2. I've been working around the need for depot_tools by teapot_tools, but there's no teapot git-cl, so I must figure this out one way or another
selfisekai commented 1 year ago

@sortie submitted icu changes: https://chromium-review.googlesource.com/c/chromium/deps/icu/+/4307370

selfisekai commented 1 year ago

@sortie I've tried to submit zlib changes, but I'm failing with this:

remote: PERMISSION_DENIED: The caller does not have permission
remote: [type.googleapis.com/google.rpc.RequestInfo]
remote: request_id: "3a3e232721f04ca7a161655235204c1d"
remote: 
remote: [type.googleapis.com/google.rpc.RequestInfo]
remote: request_id: "3b195fb9bc984285a14ef5ae86b4ebe2"
fatal: unable to access 'https://chromium.googlesource.com/chromium/src/third_party/zlib.git/': The requested URL returned error: 403
adenilsoncavalcanti commented 1 year ago

You may want to rethink the idea of dropping Chromium zlib.

It is a highly optimized fork of zlib using SIMD instructions on x86 and Arm.

For further info, check: https://docs.google.com/presentation/d/1_XWLhF4FcTz_HGc4QqvVapkxMj2-QTOZNdyXvpy2oYU/edit?usp=sharing https://docs.google.com/presentation/d/1-sNwEtudmblPKuoIR-Skls3j7NMO24G1BG9Si_1JWrI/edit?usp=sharing

If anything, I'm seriously considering removing support for 'use_system_zlib', as there are no good reasons (performance or security) to use something else.

To quote the discussion (https://chromium-review.googlesource.com/c/chromium/src/+/947464): "I wouldn't recommend running Chromium against a zlib that is not third_party/zlib.".

nekopsykose commented 1 year ago

dropping

i am not sure where it is suggested to "drop" it, this is a proposal for a toggle, not removal.

adenilsoncavalcanti commented 1 year ago

I can't comment on the other dependencies (icu, protobuf, tcmalloc) but what is the gain here for the zlib case?

adenilsoncavalcanti commented 1 year ago

Even if you don't care about performance, from a security perspective would be a bad idea: https://docs.google.com/presentation/d/1GNCnusHEVFY3vuqVMUh1b6fqo212T0HYkYSVe3f3CPw/edit?usp=sharing&resourcekey=0-8_xbWFtMBBLM60NoZwUtnQ

nekopsykose commented 1 year ago

the same as for icu, protobuf, tcmalloc, that it dynamically links a dependency from the system, which allows upgrades (and potential security fixes, etc) independently without requiring any rebuilds for reverse dependencies.

that foundation is why the majority of linux distributions favor dynamic over static linking (though of course there's experiments into various methods wrt all this, etc).

now you could say something like 'well, zlib updates with chromium, which itself updates fairly often, etc', but that still requires a rebuild of the entirety of chromium (on my machine this is about 4 hours without cache) if potentially someone immediately had to release a security fix for the zlib in question.

that's also assuming that 'system zlib' here is actually 'canonical zlib'. i think in the coming years we will see something like zlib-ng take priority for /usr/lib/libz.so.1 over the canonical. but you can't actually know what is there! it is just a preference for dynamic linking, not necessarily "remove the google code".

that said, if you have any zlib 0-days to make my day interesting, drop them right here :)

(this also reminds me to look into the feasibility of making zlib-ng the default in the distro i maintain..)

adenilsoncavalcanti commented 1 year ago

It seems the issue you are trying to solve is related to compilation time, perhaps Goma could help here: https://chromium.googlesource.com/infra/goma/client/+/HEAD/doc/early-access-guide.md

selfisekai commented 1 year ago

it's also a logistics issue. a list of questions comes in with a security upgrade: what packages do have zlib compiled in statically? how to upgrade the version in a codebase, if the source code of a specific version is in the tarball together with other source code? do we have to wait for each project to make a release with upgraded version?

nekopsykose commented 1 year ago

i have no idea how a chromium/google -internal tool open only to chromium contributors and supporting only *checks webpage* ubuntu 16.04, helps anyone build chromium (nor is that even the core issue), but maybe working for google makes you think everyone else has build farms of thousands of machines too?

i'll strike a deal: i'll use chromium zlib if chromium supports JpegXL.

adenilsoncavalcanti commented 1 year ago

i'll strike a deal: i'll use chromium zlib if chromium supports JpegXL.

I'm unsure what is your intention here, but it doesn't seem productive or helpful.

tiziodcaio commented 5 months ago

We should try unbundle binaryen too!