Plan to be distribution friendly

bastien-roucaries commented 4 years ago

Hi,

Two years ago you planned to be distribution friendly (https://docs.google.com/document/d/1VuYv41U_Esjrl0umu-uO3hUK4WNnCiG0pXZgjliHfTw/)

On the debian side we are dropping IA package (tensorflow) because we could not get bazel on debian.

Did you have some plan to be distrib friendly ?

Thanks

rouca@debian.org

bastien-roucaries commented 4 years ago

Note that we have outdated patch here https://github.com/kmoffett/bazel/tree/debian/sid/debian/patches

bastien-roucaries commented 4 years ago

Any plan and delay for this ?

dslomov commented 4 years ago

We are not actively pursuing getting into Debian atm. However, we would welcome community efforts.

bastien-roucaries commented 4 years ago

Le lun. 21 oct. 2019 à 10:58, Dmitry Lomov notifications@github.com a écrit :

We are not actively pursuing getting into Debian atm. However, we would welcome community efforts.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/9408?email_source=notifications&email_token=ABE5UHCYOSFSBMQVXP5EULTQPVVMFA5CNFSM4IYP6V32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBZSWII#issuecomment-544418593, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABE5UHHLN3I5HOY7NOLYPQDQPVVMFANCNFSM4IYP6V3Q .

Coule you at least write a plan for getting distri friendly with incremental step ?

bastien-roucaries commented 4 years ago

@dslomov Could you please give us some guidance to be step by step distrib friendly and upstream includable ?

I can code myself but I need plan

philwo commented 4 years ago

Hi @bastien-roucaries,

thank you for offering to help. I'll follow up in the next days with action items that we should address. I think we have some notes about what we have to do here, I'll try to find them and will post everything here. If you don't hear from me soon, please ping me here on the issue.

Some things out of my head:

The Bazel binary should not be self-extracting, instead we should package the install_base so that it's present in /usr/libexec/bazel (?) and Bazel just uses it.
Bazel should not use an embedded JDK, but the one provided by the distro by default.
We have to figure out how to deal with dependencies - we probably have to build Bazel against third-party libraries provided by the distro, instead of the versions in our third_party folder.

olekw commented 4 years ago

Hi @philwo,

Ping. :) I'm working with the Debian Med team right now to get some much-needed software packaged and available for users in the medical community to help with the COVID-19 pandemic. At least one of the packages we desperately need requires Bazel to build.

Clearly this is an unusual and very critical situation. I don't think it's an exaggeration to say that lives may literally depend on us getting better tools to the medical professionals out there, and quickly.

I appreciate that you likely have your own struggles right now but there are a number of us who are willing to assist. The entire international community would be extraordinarily grateful if @google and the @bazelbuild team could prioritize helping with this!

Thanks in advance and I'm waiting and eager to help however I can!

-Olek

olekw commented 4 years ago

Also, it probably goes without saying but we'd love to be able to highlight @google support when we summarize the impacts of the international effort towards the COVID-19 Biohackathon! All of us doing a little can make a huge difference for people out there!

meisterT commented 4 years ago

Hi @olekw,

while I understand the desire to move this issue forward, please understand that it is major work which will take a lot of time to complete. And even after making Bazel distribution friendly, it will take some time to get it into the stable debian distribution.

If you need something in the short term, consider using the debian packages that we already provide (https://docs.bazel.build/versions/master/install-ubuntu.html) or use bazelisk (https://github.com/bazelbuild/bazelisk). Let us know if any of those help you to satisfy your current needs.

Tobi

olekw commented 4 years ago

@meisterT,

Thank you for your reply. However, I'm afraid neither of the options you presented can be packaged in Debian and therefore the medical packages we need will also not be able to be included in Debian. It's not that an individual user can't install Bazel itself, it's that we're trying to build critical medical packages in Debian using Bazel. And we are targeting Debian Unstable at the moment, not Stable. I hope that clarifies what we're trying to do.

To be clear, we're moving forward on making @bazelbuild work in Debian. This is too important to just wait. But we would really love to have support from the upstream team and from @google in this effort!

meisterT commented 4 years ago

Thanks for this clarification. May I ask which package(s) you want to build with Bazel?

Probably the first thing one should do to move this issue forward is to go over our (vendored) third party dependencies (see https://github.com/bazelbuild/bazel/tree/master/third_party) and for each

check whether we can drop it,
and if dropping is infeasible check if there's already a debian package that provides the dependency (in ~the right version).

olekw commented 4 years ago

Hi @meisterT,

The highest-priority package we're trying to add to Debian is TensorFlow (https://salsa.debian.org/science-team/tensorflow).

Thanks! Yes, I've been working on that. My personal perspective (I'm a DD but not on the FTP Master team) is that we could leave certain third-party dependencies bundled. Here's how I see that: 1) Dependency is in Debian and usable: Drop and use Debian version 2) Dependency is in Debian and not usable: Update Debian version so it works with Bazel 3) Dependency is not in Debian and licensing allows it to be included in the Bazel source package: leave it in (if other programs need to use that dependency in the future we could always spin it off into its own package) 4) Dependency is not in Debian and licensing does not allow it to be included in the Bazel source package: work on separately packaging (although I'm not sure if any dependencies fall into this category)

I would love your inputs as to which dependencies could theoretically be dropped (and the impact of dropping them) so that we don't spend tons of time on something that doesn't appreciably impact Bazel functionality. I've looked through the old Dependencies Google Sheet, and the Debianization Google Doc although those seem old and I'm not sure how updated they are. I might recreate those on salsa.debian.org so that we can have everything together and update as needed.

hicksjoseph commented 4 years ago

Hi Olek, I am the Product Manager for Bazel. I have sent you an email with my contact information so that we can have a quick call so that I can fully understand your needs and the urgency of your request. Please feel free to contact me any time via the contact methods in my email to discuss this directly. Thanks, Joe

Ericson2314 commented 4 years ago

With supreme difficulty the NixOS distro has managed to package software built with Bazel, including tensorflow right https://github.com/NixOS/nixpkgs/blob/29eddfc36d720dcc4822581175217543b387b1e8/pkgs/top-level/python-packages.nix#L6561-L6623 . If our code, or our explaining of code could be of some assistance, do let us know.

Bazel really insists on wanting to control the world like only a distro should, so there is still rather more vendoring of random things here than we would like, but it does build and run.

Ericson2314 commented 4 years ago

For the long term beyondn Covid-19, I would recommend following the lead of Meson with their "wrap dependencies" https://mesonbuild.com/Wrap-dependency-system-manual.html . Basically, the purpose of unblocking devs on all platforms with vendoring is laudible, but it is also synonymous with being a package manger. So own it and be a de jure not de facto package manager, and share the vendoring code between projects while one is at it. But make all that vendoring optional so there is a vendor-nothing version of the build distros can use.

[Full disclosure, I have contributed a bunch to Meson, so perhaps I am biased. But I began contributing only long after I already was packaging things for Nixpkgs/NixOS, and was inspired to do so because it was one of the only new build systems which did play nice with distros.]

hicksjoseph commented 4 years ago

The Bazel team spoke to Olek and others and are working together on an assessment of work and plan per the request above. Ping this thread if you have questions / concerns.

olekw commented 4 years ago

Hi @Ericson2314 and thanks for the extra info! I'm not very familiar with NixOS therefore I'm not sure how much of your work we'd be able to reuse but I'll definitely take a look. I appreciate the pointer!

Ericson2314 commented 4 years ago

@olekw Nixos/Nixpkgs is quite different from Debian, but one ignore all that and just look at the bash templating, which should be useful prior art for any disto cherry-pick. I am happy to link the relevant bits so you don't have to search for them if you like. [I linked roughly a list of the tensorflow-related packages we had, separate from the details of any of the,.]

olekw commented 4 years ago

That would be great, @ericson2314! Would also be helpful for other downstream developers who may be following this thread.

aiuto commented 4 years ago

I don't characterize this as being a package manager at all. The end result of building Bazel is a single binary. If we use libfoo-1.7.2 to build it, that does not mean that the OS distribution we are building for also must use or provide libfoo-1.7.2, nor that any users of Bazel are bound to that version. The vendored in packages are just more source of Bazel.

Now, you might not agree with that stance and might take an opposing position that you must build tools for distribution X only from your own source plus packages already provided for distribution X. Sometimes that is easy. Sometimes not, because the required packages are simply not available for that distribution. Getting to vendor-nothing will be extremely costly for some of our oddball dependencies.

On Mon, Apr 13, 2020 at 11:08 AM John Ericson notifications@github.com wrote:

For the longer term than Covid-19, I would recommend following the lead of Meson with their "wrap dependencies" https://mesonbuild.com/Wrap-dependency-system-manual.html . Basically, the purpose of unblocking devs on all platforms with vendoring is laudible, but it is also synonymous with being a package manger. So own it and be a de jure not de facto package manager, and share the vendoring code between projects while one is at it. But make all that vendoring optional so there is a vendor-nothing version of the build distros can use.

(Full disclosure, I have contributed a bunch to Meson, so perhaps I am biased. But I began contributing only long after I already was packaging things for Nixpkgs/NixOS, and was inspired to do so because it was one of the only new build systems which did play nice with distros.)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/9408#issuecomment-612939056, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXHHHFELGBLICGLYQJYE63RMMTGLANCNFSM4IYP6V3Q .

Ericson2314 commented 4 years ago

@aiuto I would never ask any project to restrict itself from using different versions or customized or patched versions of dependencies. But, I do ask that if a distribution/package manager were to provide those packages at those versions, the project can use the distribution-/package-manager-provided builds instead of it's own vendored binaries/sources.

[With NixOS there is no cannonical version of a package, so we have no technical problem patching upstream things for downstream packages. (In fact we have 3 versions of bazel in order to support tensorflow 2.1 and 1.0, and provides a >=1 version of bazel for other users. All can be installed concurrently.) Our readiness to gave a package anything it could possibly want makes mandatory vendoring especially irksome.]

Ericson2314 commented 4 years ago

@olekw

One of the bazel versions: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel/bazel_0_29/default.nix
Function to help template strings for bazel packages (the function despite its name doesn't do any building, it just does templating) https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/build-bazel-package/default.nix
One of the tensorflow versions, which uses buildBazelPackage: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/tensorflow/1/default.nix

olekw commented 4 years ago

Update: the Debian Bazel team has expanded our workplan. Phase 2 will focus on helping to resolve this issue. We invite anyone with Debian experience or interest to join the team in any capacity. Contributing code is great, commenting and providing ideas is appreciated as well. We'll continue to post periodic updates here as we make progress. Hoping to bring a shiny native Bazel binary to a Debian (or derivative) system near you soon! :)

davido commented 4 years ago

These efforts were mentioned in my PR https://github.com/bazelbuild/bazel/pull/11271#issuecomment-623315162 to bump Error Prone version to 2.3.4 in Bazel. Recent EP release added dependency on Ben-Manes Caffeine cache library: [1].

@ben-manes Can you comment on current state of Caffeine package in Debian? I have only found outdated package: [2].

[1] https://github.com/ben-manes/caffeine [2] https://debian.pkgs.org/sid/debian-main-arm64/libcaffeine-java_2.6.2-1_all.deb.html

ben-manes commented 4 years ago

This was packaged by the Debian Java Maintainers and you would have to contact them.

olekw commented 4 years ago

@ben-manes Can you comment on current state of Caffeine package in Debian? I have only found outdated package: [2].

@davido, if you need a newer version in Debian I recommend that you file a bug against the source package "caffeine-cache" [1] requesting that they upgrade to the newer version and explaining the impact to you of not having the new version.

[1] https://packages.debian.org/source/sid/caffeine-cache

olekw commented 4 years ago

I would just like to point out that #11281 is HUGE and goes a very long way towards enabling the ability of distributions to easily ship individualized versions of Bazel that will use native libraries. I encourage anyone following this issue to examine that commit if you haven't already. #11298 extends that ability from Java dependencies to C++ dependencies.

olekw commented 4 years ago

For anyone following this issue: we have made great strides in the past several weeks and I'm excited to continue. However, we've run into a number of Debian dependencies that are not packaged and that has significantly slowed down the effort. (I'm working on packaging dependencies vs preparing the Debian Bazel package)

So, if you use Debian, Ubuntu, or any of our derivatives: we could really use your help! Any assistance at all would be appreciated and this will help you get these capabilities sooner. Please check our wiki page and see if there's anything that you're able to help with!

If you don't use a Debian-based distro, this applies to you as well. We've been very careful to make this framework extensible so everyone can easily use and adapt it for their distros. I'm very excited for everyone to get this, please help us make that sooner than later!

Thanks in advance!

swankd commented 4 years ago

@hicksjoseph @google May I ask if there is any news on this issue and if any assistance is needed? I would be happy to help!

olekw commented 4 years ago

@swankd we've been making a lot of progress on the initial proof of concept with Debian. When that is working well, it will be fairly easy to extend it to other distributions. We welcome you to join us on the Debian Bazel team[1] to get everything ironed out quickly!

[1] https://salsa.debian.org/bazel-team/meta/-/wikis/Workplan-Part-1

davido commented 4 years ago

So, Error Prone upgrade was submitted, that depends on Caffeine Cache. However, it's outdated on Debian. I figured, to contact Java Maintainers the email address is: pkg-java-maintainers @ lists.alioth.debian.org. But would it be enough to kindly ask to upgrade Caffeine Cache version?

https://packages.debian.org/source/sid/caffeine-cache

olekw commented 4 years ago

Hi @davido. The best way of requesting a package upgrade in Debian is by submitting a bug report. The reportbug program is the easiest way of doing so if you're on a Debian-based OS.

However, can you give me some more info since I happen to the packager/maintainer for Error Prone in Debian? I currently have Error Prone 2.4.0 in Debian unstable. That version depends on Caffeine 2.8.0 which is the same requirement as HEAD. So I don't see an issue there. Or do you mean that you would like the Error Prone package to include additional modules? I currently only build annotation, annotations, and type_annotations. The others have dependency issues and, as far as I'm aware, are not required by Bazel.

davido commented 4 years ago

@olekw See: https://github.com/bazelbuild/bazel/pull/11426#issuecomment-651066064.

third_party/caffeine/caffeine-2.8.0.jar: OK
third_party/checker_framework_annotations/checker-qual-3.2.0-sources.jar: OK
third_party/checker_framework_annotations/checker-qual-3.2.0.jar: OK
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2-sources.jar: OK
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2.jar: OK
third_party/checker_framework_javacutil/javacutil-3.2.0-sources.jar: OK
third_party/checker_framework_javacutil/javacutil-3.2.0.jar: OK
third_party/error_prone/error_prone_annotation-2.4.0.jar: OK
third_party/error_prone/error_prone_annotations-2.4.0.jar: OK
third_party/error_prone/error_prone_check_api-2.4.0.jar: OK
third_party/error_prone/error_prone_core-2.4.0.jar: OK
third_party/error_prone/error_prone_type_annotations-2.4.0.jar: OK
third_party/error_prone/threeten-extra-1.5.0.jar: OK

Particularly, these were new dependencies in Bazel:

third_party/caffeine/caffeine-2.8.0.jar
third_party/checker_framework_dataflow/dataflow-shaded-3.1.2.jar
third_party/error_prone/threeten-extra-1.5.0.jar

Are these libraries available in Debian unstable?

davido commented 4 years ago

OK, answering my own question:

You have uploaded Error Prone upstream code base 2 weeks ago to Debian Salsa, and the new EP transitive dependencies, I mentioned in my previous comment seems to be there:

https://salsa.debian.org/java-team/error-prone-java/-/blob/master/check_api/pom.xml#L65 https://salsa.debian.org/java-team/error-prone-java/-/blob/master/core/pom.xml#L356

And I also found:

https://tracker.debian.org/pkg/threeten-extra https://salsa.debian.org/java-team/checker-framework-java/-/tree/master/dataflow

olekw commented 3 years ago

Hi @davido. So those artifacts are not currently built in the Debian package since I was trying to keep the new package as simple as possible and only include the minimum needed for Bazel. As I mentioned earlier, Error Prone only builds three artifacts in Debian and Checker Framework (once it clears the NEW queue) will only build checker-qual and checker-qual-android.

Please let me know right away if additional artifacts are mandatory for Bazel to build! Last I checked with @meteorcloudy, the currently-packaged artifacts were the only ones we needed. For all of the Bazel dependencies that I packaged in Debian I typically added additional (not required by Bazel) artifacts if there were no dependency problems with including them. Building additional artifacts will likely require software not currently packaged in Debian and that will likely add significant time and effort to this project.

sgowroji commented 1 year ago

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

olekw commented 1 year ago

@sgowroji, no, since this has not yet been completed I do not think it should be closed. This is still an ongoing effort and therefore this bug should remain open to track that effort.

meteorcloudy commented 1 year ago

Bazel's third_party directory is much smaller now since we have migrated all checked-in jar dependencies to rules_jvm_external, which should make it easier to package Bazel for Linux distributions in future (for example, we only need to override the @maven repository to make use of existing jar files under /usr/share/maven-repo on Debian)

meisterT commented 1 month ago

@olekw What are the main blockers from your POV to make Bazel distribution friendly?

bazelbuild / bazel

Plan to be distribution friendly #9408