pip install drake - Githubissues

RussTedrake commented 9 years ago

Was titled pip install drake; apt-get install drake; brew install drake. See #12782 for brew install drake and #12783 for apt-get install drake.

peteflorence commented 9 years ago

this would be awesome!

On Wed, Jul 15, 2015 at 2:35 PM, Russ Tedrake notifications@github.com wrote:

make debian packages, etc.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1183.

jhoare commented 8 years ago

A good way to support apt-get for Ubuntu is to run a drake PPA. You could have a stable/nightly/etc versions where users could automatically get updates via apt-get update.

https://help.launchpad.net/Packaging/PPA

RussTedrake commented 8 years ago

while I’m sure there are many examples, I found this page to be very compatible to what I think we should strive for here: http://dreal.github.io/download/

david-german-tri commented 8 years ago

I think this is blocked on stable Drake releases, which in turn is blocked on #1769.

jamiesnape commented 6 years ago

Note drake is already taken as a name in homebrew/core. I was just trying out building a Homebrew formula. Looks like a few complications relating to pkg-config and the Homebrew environment, nothing that cannot be fixed, though. I can already build an apt package.

jamiesnape commented 6 years ago

I have a Homebrew package that builds, but we have conflicts with the eigen, bullet, fmt, spdlog so it will not link.

jwnimmer-tri commented 6 years ago

Possibly pip install pydrake is also a useful offering.

jamiesnape commented 6 years ago

Indeed. We have a lot of experience with that. (You can pip install cmake itk vtk).

EricCousineau-TRI commented 6 years ago

Regarding pip, per Jamie's comment in #9456 and when talking with him, it'd be nice to keep this on low-priority, esp. with regards to distribution (e.g. making it such that pip install drake does not have any dependencies on system packages).

Ideally, it'd be nice to support one pip environment, and thus wait until we transition to just using Python3.

RussTedrake commented 6 years ago

That would make #9456 all the more important. My goal is simply to make it easy to use/install. apt-get/brew is another good solution, as far as I'm concerned. But even downloading the tarball might be ok if the workflow is not so onerous.

EricCousineau-TRI commented 5 years ago

Merging #7982 here, "Installed locations of vendored dependencies conflict with Homebrew installed versions":

Namely eigen, bullet, fmt, and spdlog. Following Homebrew conventions, we could install to libexec/vendor/{include|lib} or we could even just use system versions.

RussTedrake commented 4 years ago

Can we revisit this now (we're python3 only)? "Why don't we support pip install drake?" (or pydrake) is the single most common question I've been getting lately. (fwiw -- it's pip install torch to get pytorch)

jamiesnape commented 4 years ago

The reason is not to do with Python versions, but rather the heavy dependence on dynamically linking system dependencies. We can do it, but it changes/disrupts the C++ core significantly.

jamiesnape commented 4 years ago

(I can give you brew cask install drake straight away, however, and apt-get install drake is do-able if you install to /opt rather than /usr)

jamiesnape commented 4 years ago

Note, there is https://github.com/pypa/auditwheel, but only handles a subset of the issues (https://www.python.org/dev/peps/pep-0599/ is also instructive).

EricCousineau-TRI commented 4 years ago

I think we could potentially revisit, but I would still like to defer this until after we have the bindings automation (#7889) underway, given that this would require a hefty bit of dependency rewiring. Would that be OK?

FTR, I'd be down for slotting apt-get and brew for the near-term, given Jamie's eval on it being less overhead.

RussTedrake commented 4 years ago

i think apt-get and brew would be a huge improvement!

jamiesnape commented 4 years ago

You are welcome to try out:

brew tap jamiesnape/drake
brew cask install drake

Needs more testing and some work to make it maintainable.

jamiesnape commented 4 years ago

I took a quick look at the apt packaging and pending on how important this is, it is looking like 3-5 days work for manual packaging of the latest numbered release (subsequent releases would be a few hours). You would get apt-get install python3-drake libdrake1 libdrake-dev libdrake-java at a minimum. Full automation would take longer. Most of the task is actually administrative, rather than technical.

Really, we nee a test suite for the packages that I can run during the builds (something like a script that extracts a few files from drake/examples and writes both a CMakeLists.txt and {BUILD,WORKSPACE}.bazel. drake-external-examples is not sufficient for that purpose (and nor was it supposed to be).

jamiesnape commented 4 years ago

Can we split this into either two issues (apt and brew cask one issue) or three? pip is so much more complicated that this issue is going to be two thirds complete for a very long time.

EricCousineau-TRI commented 4 years ago

Can we split this into either two issues (apt and brew cask one issue) or three? [...]

Yes, please!

jamiesnape commented 4 years ago

Done. Let's keep this as pip install drake since I think there is some useful information here.

RussTedrake commented 4 years ago

@jamiesnape -- when we talked about this last week, you further explained your concern about the linking issues (and how it would disrupt the c++ core). Basically, you're worried that pip means you have to provide the same experience for all users on all platforms, and the "disrupt the c++ core" was because you thought it implies that we have to support redhat, etc in the main repo.

i don't think we should do that. but my ask for you was whether we can support only a subset of platforms with pip. E.g. pip users either get pydrake, or they get an unsupported platform if they are not on ubuntu or mac. where you able to look into that?

And btw -- i'm ok if we support ubuntu only in the first pass.

EricCousineau-TRI commented 4 years ago

Huh - I thought the "disrupt c++ core" would've implied moreso that you have to do what TensorFlow / PyTorch do - bake all non-Python dependencies into this binary, and have next to zero reliance on the system's package manager; not so much the platform support side of things.

Had y'all already discussed that?

jwnimmer-tri commented 4 years ago

Any technical exposition, details, or work planning debates from realtime meetings should be transcribed into the issues database (or incorporated via reference to a separate document). It sounds like that has not been happening for the Russ <=> Jamie meetings.

jamiesnape commented 4 years ago

Huh - I thought the "disrupt c++ core" would've implied moreso that you have to do what TensorFlow / PyTorch do - bake all > non-Python dependencies into this binary, and have next to zero reliance on the system's package manager; not so much the platform support side of things.

Had y'all already discussed that?

It is both, to be honest, though there are a few more ways to solve the issue if you know what platform you are on.

Any technical exposition, details, or work planning debates from realtime meetings should be transcribed into the issues database (or incorporated via reference to a separate document). It sounds like that has not been happening for the Russ <=> Jamie meetings.

I do not think I have said anything that has not been written elsewhere (and the meetings were minuted in the usual location in any case).

jamiesnape commented 4 years ago

FWIW This is how platform compatibility tags work: https://packaging.python.org/specifications/platform-compatibility-tags/. As I have said, you are limited to flavors of manylinux. manylinux2014 was not approved until 2019, so any approval of a more up-to-date manylinux is looking to be years away, if ever (the 2014 process caused enough issues that things will probably change). Since drake requires C++17, it would need to fall back to C++17 polyfills at the very least.

ToffeeAlbina-TRI commented 4 years ago

Jamie, Jeremy, and Eric still believe pip is much more complicated, risky and expensive than apt. It’s not worth pursuing for the September timeframe - too risky.

jwnimmer-tri commented 3 years ago

I'm moving this into "On Deck" so that we can start planning out the work.

ludwigschmidt commented 3 years ago

Yay! Looking forward to pip install drake!

jwnimmer-tri commented 3 years ago

Next step on deck, per chat with @jamiesnape:

In our prototyping repo pip-drake, we'll work out a 18.04-based Dockerfile that runs the Drake install, and then at the end we'll show a report with the problematic shared libraries, and then we can iterate to remove system dependencies and turn them into source dependencies.

jwnimmer-tri commented 3 years ago

Looking at a current (v0.30.0) binary release tarball in the drake/lib/python3.6/site-packages folder, anything that's not part of the pydrake module could present a problem for a pydrake pip wheel. (Everything in the wheel must be within the pydrake module).

Here's the ones that are easily resolved:

A bunch of stuff from drake-visualizer (director, urdf_parser_py, ...). No need for this to be in the wheel, nothing depends on it. We can either remove it post-hoc, or add an install knob to disable it up front.
meshcat. We'll declare a dependency on it from our wheel, so that pip will pull it in first-party. We can either remove it post-hoc, or add an install knob to disable it up front.
The soon-to-be-removed bot_core and robotlocomotion lcmtypes. These will evaporate soon anyway.

These lcm messages remain a problem:

drake.lcmt_... lcmtypes
optitrack.optitrack_..._t lcmtypes

My goal for drake.lcmt_foo is to respell it as pydrake.lcmtypes.lcmt_foo for all users (not just wheel users) across the board.

For optitrack python messages, from a first quick look I don't see pydrake importing them at all, so for a first pass we could just drop them from the wheel.

Also, there is:

lcm (module with a native shared library)

I haven't looked into whether this presents any problem, or what to do about it yet.

jwnimmer-tri commented 3 years ago

lcm (module with a native shared library)

On my very first skim, I don't see any code in pydrake that actually imports LCM. I think we were just installing it as a courtesy to downstream users, so that they have a python lcm that reuses the same shared library as us.

Therefore, it's likely to be a non-issue, at least for the initial release(s).

jwnimmer-tri commented 3 years ago

We have https://pypi.org/project/drake/ published now (as of https://github.com/RobotLocomotion/drake/releases/tag/v0.35.0). Thanks to everyone who worked hard to make that happen!

The wheels have some known problems as noted in https://github.com/RobotLocomotion/drake/issues/15954. Some of those will be fixed for the next v0.36.0 release, which will be coming out next week or so, and we'll continue to work on fixing the others. Please upvote or reply in those issues if there is anything particularly painful, or open a new issue for problems that aren't mentioned there yet.

At this point, I don't think this meta-issue tracking is adding any value, so I'm going to declare this "done", and we'll use the more fine-grained issues to manage our progress.

RobotLocomotion / drake

pip install drake #1183