Build drake in a Debian Bookworm release

[lepalom]

I would like to build drake in a Debian 12 (Bookworm). As Ubuntu is a derivative of Debian, it should be smoothly, however it is being a pain.

I have modified: CMakeLists.txt tools/workspace/os.bzl tools/workspace/python/repository.bzl and added all the Debian 12 entries that I have found. However, building drake_cxx_python it fails with:

ERROR: An error occurred during the fetch of repository 'python': Traceback (most recent call last): File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/python/repository.bzl", line 133, column 37, in _impl py_info = repository_python_info( File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/python/repository.bzl", line 68, column 49, in repository_python_info versions_supported = _VERSION_SUPPORT_MATRIX[os_key] Error: key None not found in dictionary

in that file in _VERSION_SUPPORT_MATRIX I have the key: "debian:12": and "debian": however, it seems that if fails downloading something?

So, I would like to ask:

1) There's a more clean way to build drake in a more agnostic Unix way? 2) To build drake, it is necessary to download files? 3) Which are the steps needed to add a platform in drake?

Thanks

[jwnimmer-tri]

As the Drake build system maintainer, I can give some answers.

Background: It's been a long, ongoing project (#14967) to improve Drake's compatibility with Linux systems that are not Ubuntu. We are much closer to the goal this year than at any time prior, but it's not 100% finished yet. In terms of what we claim to support on our website, that will always be driven by what's in our Continuous Integration test matrix. If we aren't testing it in CI, we won't claim to support it. However, if we're careful about how we set up the build and dependencies, it can still be true that building on other Linux distributions could be smooth.

---

Before answering your questions, let me try to provide some alternative work-arounds to get you up and running for now.

(1) Each release like https://github.com/RobotLocomotion/drake/releases/tag/v1.23.0 comes with pre-compiled *.deb packages. They are only built for Jammy and Focal, but there's a real chance the Jammy one would just work for you. I spot-checked a few of its Dependencies shared libraries, and all of the same ABI libraries were part of Bookworm.

(2) The release also has a *.tar.gz file with the binaries, which you could unpack and run directly, in case the *.deb refused to install for some spurious reason.

---

Now to your questions, related to building from scratch from source ...

The Python repository crash was just a bug. Our aim there is to let your CMake variables dictate which Python interpreter to use, but we never tested that new logic on Debian. I'll filed a proposed fix at #20562. (If you're able to test it, please post in that pull request with any feedback.)

In terms of modifying os.bzl, it's primary objective is to specify / disambiguate these choices:

• Whether we are (only) building wheels for PyPI, or a full release.
• Whether we're building on Mac or Ubuntu.
• Whether we're using Ubuntu 20.04 Focal (which is old enough to require some hacky special-case workarounds in the build system).

Therefore, adding a totally-new option for "Debian" there is probably going to be more pain than its worth right now. Instead, I advise changing it to claim that it's using Ubuntu 22.04 (not Debian Bookworm). That provides the right answer for the prior three questions (full release, not Mac, no hacks) without stepping too far off the beaten path. In the future, I can imagine that the second question should be "Whether we're building on a Mac Homebrew system or a FHS-based system" but that's a somewhat larger refactor.

To build drake, it is necessary to download files?

I'm not sure exactly what you're asking here.

The Drake build process itself automatically downloads a lot of external source packages and compiles them.

Or if you're asking about what needs to exist on the system already, the answer is there the install_prereqs.sh script which ends up running apt to install a few groups of packages, e.g., source_distribution/packages-jammy.txt. (You would probably run it with --without-test-only if you don't plan to run any tests.)

[lepalom]

@jwnimmer-tri thanks for the answer. Just some points. I'm a bit confuse about why be so specific with the Ubuntu version. IMHO it is more important to take care about the versions of the dependencies. If some distro, or OS provide a version of libfoo it is not important. The important thing is the version of libfoo, not that Ubuntu 20 provides libfoo x.y and Ubuntu 22 libfoo(x+1.y+2). But it is your project, I would like to focus to run drake in our debian system and maybe, package it for Debian, under the Debian Science team.

About the alternative workaround. I understood that you provided a close form software ([SNOPT] as a solver. drake can work only with free software?

I have tested #20562 but I still have an error:

If the definition of 'rules_python' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'rules_python':
   Traceback (most recent call last):
    File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/github.bzl", line 120, column 37, in _github_archive_real_impl
        result = setup_github_repository(repository_ctx)
    File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/github.bzl", line 199, column 14, in setup_github_repository
        patch(repository_ctx)
    File "/home/users/leopold.palomo/.cache/bazel/_bazel_leopold.palomo/eaf8ef9d4722a8a12a9381a757d7d496/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 140, column 37, in patch
        patchfile = ctx.path(patchfile),
Error in path: Unable to load package for @drake//tools/workspace/rules_python:patches/internal_config_repo.patch: BUILD file not found in directory 'tools/workspace/rules_python' of external repository @drake. Add a BUILD file to a directory to mark it as a package.
ERROR: Error fetching repository: Traceback (most recent call last):
    File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/github.bzl", line 120, column 37, in _github_archive_real_impl
        result = setup_github_repository(repository_ctx)
    File "/home/users/leopold.palomo/robotica/projects/drake.git/tools/workspace/github.bzl", line 199, column 14, in setup_github_repository
        patch(repository_ctx)
    File "/home/users/leopold.palomo/.cache/bazel/_bazel_leopold.palomo/eaf8ef9d4722a8a12a9381a757d7d496/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 140, column 37, in patch
        patchfile = ctx.path(patchfile),
Error in path: Unable to load package for @drake//tools/workspace/rules_python:patches/internal_config_repo.patch: BUILD file not found in directory 'tools/workspace/rules_python' of external repository @drake. Add a BUILD file to a directory to mark it as a package.
ERROR: Skipping '//:install': no such package '@rules_python//python': Unable to load package for @drake//tools/workspace/rules_python:patches/internal_config_repo.patch: BUILD file not found in directory 'tools/workspace/rules_python' of external repository @drake. Add a BUILD file to a directory to mark it as a package.
WARNING: Target pattern parsing failed.
ERROR: no such package '@rules_python//python': Unable to load package for @drake//tools/workspace/rules_python:patches/internal_config_repo.patch: BUILD file not found in directory 'tools/workspace/rules_python' of external repository @drake. Add a BUILD file to a directory to mark it as a package.
INFO: Elapsed time: 3,137s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: 

And about the question to download files. Debian policy do not allow to download files when you build a package. If you do it, you cannot ensure the reproducibility of the package. The package must be built from one source, and it is highly recommended (almost mandatory) to use the libraries packaged, not to use your patched version of the library that you use.

So, can I build drake without download any software, only build from the sources you provide?

[jwnimmer-tri]

I have tested https://github.com/RobotLocomotion/drake/pull/20562. but I still have an error: ...

Please discuss that in #20562 instead of here. Let me know which version of Drake you are using (if you cherry-picked the patch instead of running the exact branch), and provide the full set of command(s) you're using to run the build. I can try to reproduce it in a Bookworm virtual machine.

I'm a bit confused about why be so specific with the Ubuntu version. IMHO it is more important to take care about the versions of the dependencies. If some distro, or OS provide a version of libfoo it is not important. The important thing is the version of libfoo, not that Ubuntu 20 provides libfoo x.y and Ubuntu 22 libfoo(x+1.y+2).

Yes, I mostly agree -- for the most part, Drake is insensitive to the distribution. Of Drake's 100+ dependencies, only maybe 10% are sensitive to the OS (macOS vs Ubuntu) and only maybe 2% are sensitive to the which Ubuntu distribution is being used.

As an example, where Drake needs to be especially sensitive to e.g. Ubuntu 20.04 is with libfmt and libspdlog, which have serious packaging bugs on Ubuntu 20.04 that Drake needs to hack around.

The ongoing work of #14967 is to get both of those "distro sensitivity" numbers down to 0%. It's not finished yet. For example with the libfmt hacking, if Drake cannot determine which OS version it's using, rather than crashing it should fallback to "Assume it's not Ubuntu 20.04" and just skip the hacks.

Getting #14967 all the way finished is kind of a background project for me. If you (or anyone else) would like to accelerate it by contributing pull requests, I'd be happy to review them.

I understood that you provided a close form software (SNOPT) as a solver. Drake can work only with free software?

When building Drake from source, by default only open-source dependencies are used. As an opt-in during the build, the person building the software can choose to enable three possible closed-source solvers (SNOPT, MOSEK, GUROBI) -- there are CMake option toggles for each one. Drake works fine (though sometimes less efficiently) with all three of those solvers disabled.

For the binary releases that we provide ourselves, we have enabled SNOPT and MOSEK (but not GUROBI yet), so yes there is closed-source software in those binaries. We feel this is the best service to our users, since it allows the most users as possible to run our software quickly, without needing to take the time to build. (It takes several hours to build Drake on a laptop.) If the user does not have a license file for MOSEK, then that feature is disabled at runtime instead of at compile-time. So it's easy for users who want to use the closed-source software, and those that don't can disable it later if they didn't want it.

Debian policy do not allow to download files when you build a package.

Yes, I understand. (I studied to become Debian maintainer about 20 years ago and so learned in great depth about the tools and policies, but I never finished the process to actually find some new package I could take over. Some of my knowledge might be out of date, but I think things have been pretty consistent for the past decades.)

If you do it, you cannot ensure the reproducibility of the package.

That's not accurate. Everything Drake downloads is checksummed with sha256sum using pinned version numbers and URLs, so the build (in terms of downloads) is fully reproducible. We also mirror all of the downloads to multiple places, to reduce any risk of losing them or having a cloud outage that compromises the build. We want Drake to be used for science, for which reproducible results are paramount.

The thing we lose by downloading at runtime is the ability to easily build offline (which is the part that Debian cares about, as I understand it). In order to build offline, a user would need to provide a directory filled with all of the pre-downloaded artifacts, at which point Bazel would be able to use the local folder instead of the cloud to provide those sources. So one way forward would be to smoosh together all of the vendored dependencies into a Debian-specific source package where all of the code was copied into one place.

However, that probably doesn't matter anyway, because ...

More relevantly for Debian packaging, all of the 100+ downloads I mention above are vendored dependencies. We have essentially implemented our own package manager, where we have a lockfile of source dependency versions, and we recompile the large majority of our dependencies from source (but with hidden symbols, and Drake-specific patches and bugfixes). You can look for drake/tools/workspace/*/patches/*.patch files to see the inventory of our patching. Using vendored dependencies instead of Debian-archive dependencies is contrary to Debian policy (https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies), so any official Debian package of Drake would need to either make an exception to policy, or work to un-vendor all of those. The way to do it would be to write a new drake/WORKSPACE.bazel file from scratch (to supplant drake/WORKSPACE), which provides all of the dependencies through Debian-specific means, instead of relying out Drake's default rebuild-from-source rules. It's a neat thought experiment, but I think would be a lot of effort in practice.

In a perfect world, the Debian packages of our dependencies would be sufficiently up-to-date and contain sufficient bug fixes that we could use the Debian build of them instead of our own vendored rebuild. In Drake we tried to do that for many years before we finally just gave up. The dependencies we vendor are apparently not mainstream enough in Debian to merit prompt fixes and upgrades.

Just as one example, consider an open-source solver that Drake users, Clp. Even in sid, it is three years out of date. Packaging Drake in Debian would mean bringing all of those 100 packages up to date (and packaging any which are missing).

If you're serious about this, I could help put together a bill of materials for everything that Drake (currently) needs to build, and you could compare that against what's already packaged to try to get a handle on how big of an effort it might be.

But it is your project, I would like to focus to run drake in our debian system and maybe, package it for Debian, under the Debian Science team.

Besides all of the above difficulties, there is another problem: as research code, Drake follows a rolling release policy, with a 3-month deprecation window for removals. We do not even offer a stable API, never mind a stable ABI. In other words, we do not offer any SOVERSION. Every single shared library would need to be like libdrake-1.23.0-dev and would not be forward- nor backward compatible. So IMO, any official Debian packages would tend to bitrot very quickly, and not actually offer much benefit to Debian users. For that reason, for the moment I think the private archive documented at https://drake.mit.edu/apt.html is the best bet for users who want binaries. You might have better ideas than me here, though.

---

In related news, FYI in the next ~6 months we're aiming to ship Drake *.deb packages in the ROS buildfarm. That will be another way for users to download and install Drake. Possibly that could help some of the users you have in mind. (The ROS buildfarm also frowns on vendored source dependencies, but is much less strict -- they allow it on a case-by-case basis.)

[lepalom]

@jwnimmer-tri your comments are really long :-)

Please discuss that in https://github.com/RobotLocomotion/drake/pull/20562 instead of here. Let me know which version of Drake you are using (if you cherry-picked the patch instead of running the exact branch), and provide the full set of command(s) you're using to run the build. I can try to reproduce it in a Bookworm virtual machine.

Done, I made a fork and you can test it.

As an example, where Drake needs to be especially sensitive to e.g. Ubuntu 20.04 is with libfmt and libspdlog, which have serious packaging bugs on Ubuntu 20.04 that Drake needs to hack around ...

But in Ubuntu 22.04 are solved, so I guess they were solved first in Debian.

Getting https://github.com/RobotLocomotion/drake/issues/14967 ...

I can look it.

When building Drake from source, by default only open-source dependencies are used.

Ok, clear.

If you do it, you cannot ensure the reproducibility of the package.

That's not accurate. Everything Drake downloads is checksummed with sha256sum using pinned version numbers and URLs, so the build (in terms of downloads) is fully reproducible. We also mirror all of the downloads to multiple places, to reduce any risk of losing them or having a cloud outage that compromises the build. We want Drake to be used for science, for which reproducible results are paramount.

Ok, I agree.

The thing we lose by downloading at runtime is the ability to easily build offline (which is the part that Debian cares about, as I understand it). In order to build offline, a user would need to provide a directory filled with all of the pre-downloaded artifacts, at which point Bazel would be able to use the local folder instead of the cloud to provide those sources. So one way forward would be to smoosh together all of the vendored dependencies into a Debian-specific source package where all of the code was copied into one place.

However, that probably doesn't matter anyway, because ...

More relevantly for Debian packaging, all of the 100+ downloads I mention above are vendored dependencies. We have essentially implemented our own package manager, where we have a lockfile of source dependency versions, and we recompile the large majority of our dependencies from source (but with hidden symbols, and Drake-specific patches and bugfixes). ...

Did you try to use the vendor specific in Debian bookworm?

You can look for drake/tools/workspace//patches/.patch files to see the inventory of our patching. Using vendored dependencies instead of Debian-archive dependencies is contrary to Debian policy (https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies), so any official Debian package of Drake would need to either make an exception to policy, or work to un-vendor all of those. The way to do it would be to write a new drake/WORKSPACE.bazel file from scratch (to supplant drake/WORKSPACE), which provides all of the dependencies through Debian-specific means, instead of relying out Drake's default rebuild-from-source rules. It's a neat thought experiment, but I think would be a lot of effort in practice.

Could you guide me about how to do it? I have zero experience in bazel.

In a perfect world, the Debian packages of our dependencies would be sufficiently up-to-date and contain sufficient bug fixes that we could use the Debian build of them instead of our own vendored rebuild. In Drake we tried to do that for many years before we finally just gave up. The dependencies we vendor are apparently not mainstream enough in Debian to merit prompt fixes and upgrades.

I understand. I have made a lot of backports to solve this issue. But, IMHO to keep your own versions of software developed by others many times is not an option neither.

Just as one example, consider an open-source solver that Drake users, Clp. Even in sid, it is three years out of date. Packaging Drake in Debian would mean bringing all of those 100 packages up to date (and packaging any which are missing).

coinor-clp in stable/unstable 1.17.6 (April 2020) but the last release is 1.17.9. It is a patch level release. Do you think that it is very different? I can investigate. The package has been orphaned and needs a maintainer. If you want to take care of it, most of us will thank you. :-)

If you're serious about this, I could help put together a bill of materials for everything that Drake (currently) needs to build, and you could compare that against what's already packaged to try to get a handle on how big of an effort it might be.

Yes, I would like to look it. As it seems that we will use drake in our lab, and I'm a Debian fan I would like to check it.

Besides all of the above difficulties, there is another problem: as research code, Drake follows a rolling release policy, with a 3-month deprecation window for removals. We do not even offer a stable API, never mind a stable ABI. In other words, we do not offer any SOVERSION. Every single shared library would need to be like libdrake-1.23.0-dev and would not be forward- nor backward compatible.

I see, but I'm quite sure that this state will not be ALWAYS in this way. For my experience, the projects will arrive to some stabilization process.

So IMO, any official Debian packages would tend to bitrot very quickly, and not actually offer much benefit to Debian users. For that reason, for the moment I think the private archive documented at https://drake.mit.edu/apt.html is the best bet for users who want binaries. You might have better ideas than me here, though.

Well, probably if you have there a Debian distribution I was not writing this issue ;-) In any case, I agree with you that some packages need to update system much more faster that the Debian official way. I think that ideas as fasttrack. And yes, in general, the users wants binary packages ready to use.

[lepalom]

I have built your bookworm branch. In docker it works well, however, I have to modify some paths in my bookworm station.

Also, bazel package must be removed because if not it doesn't build.

[jwnimmer-tri]

Did you try to use the vendor specific in Debian bookworm?

Sorry, I didn't understand this part.

I think that ideas as fasttrack.

Ah, interesting! I had never come across that before. It seems like a good idea.

coinor-clp in stable/unstable 1.17.6 (April 2020) but the last release is 1.17.9. It is a patch level release. Do you think that it is very different?

I am not sure. The point is that Drake only tests against the one version (the latest version) so any number of things could be wrong with the older version. At least, there seem to be symbol visibility bug fixes and performance improvements in the diffs between the two.

The point isn't that just that one package is a little bit (or a lot) behind. The same questions will come up for dozens of other libraries that Drake uses which are behind the times in sid. I'm not sure how much effort it will take on your part to upgrade all of the dependencies to new-enough versions -- it might be quick work, but I'm worried it will be a lot more work than you've anticipated.

The way to do it would be to write a new drake/WORKSPACE.bazel file from scratch (to supplant drake/WORKSPACE), which provides all of the dependencies through Debian-specific means, instead of relying out Drake's default rebuild-from-source rules. It's a neat thought experiment, but I think would be a lot of effort in practice.

Could you guide me about how to do it? I have zero experience in bazel.

I think it will be a lot of effort, and I don't really have time available to help. Some places you could look for examples are the drake/CMakeLists.txt logic that handles eigen fmt, and spdlog -- those three are (able to be) configured by CMake, and then linked in as part of the Bazel build. See also anything in drake/tools/workspace/... that calls pkg_config_repository in order to find the external by calling pkg-config.

Also, bazel package must be removed because if not it doesn't build.

Yes, Drake is only tested against Bazel 6.4, and requires a minimum of Bazel 5.1. There's some debian discussion here about the older 4.x version in Bookworm.

Drake's CMake logic to locate bazel searches for a program named bazel or failing that bazelisk. The problem is that Debian has a too-old bazel on the path at /usr/bin/bazel instead of e.g. /usr/bin/bazel-4.

I could help put together a bill of materials for everything that Drake (currently) needs to build, and you could compare that against what's already packaged to try to get a handle on how big of an effort it might be.

Yes, I would like to look it. As it seems that we will use drake in our lab, and I'm a Debian fan I would like to check it.

Here a rough draft: https://gist.github.com/jwnimmer-tri/ad2eabbaaf04786386d51a8ea5f29729. This is a list of things Drake downloads at compile time as part of the minimal build (i.e., just installing Drake, not running any tests). It's probably 95% accurate. Dependencies that are obvious already in Debian (GCC, glib, etc.) are not listed.

[jwnimmer-tri]

Many PRs have landed over the past 6 months that might make this easier.

Feel free to post new comments here, but I don't think there's any action on the Drake part, so I will close as "not planned" for now.