search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.37k stars 1.27k forks source link

Re-distribute Gurobi in Drake binary releases #10804

Open RussTedrake opened 5 years ago

RussTedrake commented 5 years ago

Edit: The original issue title was "Can users with a gurobi license (and no snopt license) use gurobi AND snopt?". We've retitled to reflect a new consensus now.

I wasn't able to find any documentation about this on our site. perhaps we can add it?

Can (or does) the binary distribution include gurobi bindings, but fail if the gurobi libraries aren't found?

I believe the alternative (having someone build from source and get snopt, despite not having the snopt source) is much harder to achieve?

@jamiesnape -- tagging you to get the current status first.

jamiesnape commented 5 years ago

No, we don't support providing your own commercial solver licenses or libraries at present. Enabling Gurobi support is currently a compile-time decision. We could add support for loading commercial solvers at runtime if available, but it would be an architectural change.

jwnimmer-tri commented 5 years ago

... and to further clarify the "add support for loading commercial solvers at runtime if available":

That would entail linking or loading solvers/gurobi_solver.pic.o, not just libgurobi80.so. Even with the delayed load of gurobi, we cannnot ship solvers/gurobi_solver.pic.o in any binary release, because it is a derived work of gurobi_c.h and we do not have (to my knowledge) permission to redistribute derived works of this form.

RussTedrake commented 5 years ago

i understand. thanks.

just to collect all of these thoughts in one place... the other alternative, I would think, is to have gurobi_solver.o always in the binary release, and have it check the license file (instead of loading the obj file) to confirm availability. would this be better? (conditional on us obtaining permission for redistributing it)

jamiesnape commented 5 years ago

Even with the delayed load of gurobi, we cannnot ship solvers/gurobi_solver.pic.o in any binary release, because it is a derived work of gurobi_c.h and we do not have (to my knowledge) permission to redistribute derived works of this form.

My reading of EULA suggests that part would not necessarily be a problem, but someone would need to check for any (other) constraints.

jamiesnape commented 5 years ago

just to collect all of these thoughts in one place... the other alternative, I would think, is to have gurobi_solver.o always in the binary release, and have it check the license file (instead of loading the obj file) to confirm availability. would this be better? (conditional on us obtaining permission for redistributing it)

I think we probably want a license check as well regardless to be nice to the end-user.

jwnimmer-tri commented 5 years ago

@RussTedrake The easiest solution along those lines would be if we have permission to redistribute both the gurobi80.so library as well as our object code derived works of gurobi_c.h such as solvers/gurobi_solver.pic.o, subject to Gurobi's default built-in license-checking. In that case, we can just enable it in binary releases by default with minimal code changes -- I guess just check the license prior to ChooseBestSolver making a decision, and tuning up any install glue code.

If we only have permission to redistribute solvers/gurobi_solver.pic.o, then we cannot unconditionally link gurobi80.so in the default release -- so either we'd need to make a new binary release flavor, or split out a -ldrake_solvers_gurobi mini-library, or use dlopen, or etc.

jamiesnape commented 5 years ago

My thoughts were along the lines of a dlopen of gurobi80.so, but certainly plenty of reasons to want to avoid that, if at all possible.

EricCousineau-TRI commented 5 years ago

Just to check: Can I ask what situation this accommodates, or what use case? Is this to check alternatives against SNOPT (given the current version / config ping pong that we're playing)?

jwnimmer-tri commented 5 years ago

My impression of the user story was "Most people are happy to just use Drake binary releases. If they have a (site) license to Gurobi, it would be nice if that Just Worked out of the box. The need is especially acute for users without a site license to SNOPT -- in that case, even if they compile from source in order to use Gurobi, they then lose the ability to use SNOPT".

RussTedrake commented 5 years ago

correct. and that happens to account for most of the academic world, as gurobi is free for academics, but SNOPT is not.

tri-ltyyu commented 5 years ago

Use case understood. Is there a research group/class actively blocked by this to gauge timing? Wondering if there's a work around such as using Colab or asking the dept. to get a license for $600.

jwnimmer-tri commented 5 years ago

Since the next action on this issue is to see if MIT has the right to redistribute Gurobi stuff (per details above), maybe the easy answer is to reassign this issue to Russ, and he can pursue that question with MIT legal on whatever pace it requires?

jwnimmer-tri commented 5 years ago

I guess @tri-ltyyu could also own it and chase down MIT legal. All I'm really saying is that @jamiesnape isn't an appropriate assignee anymore.

tri-ltyyu commented 5 years ago

Happy to chase someone down to find out, but want to confirm if it's MIT legal or Gurobi legal that we want to get permission from? From Gurobi's EULA, I see the following, which makes me think we need permission from them first? "You may not use, copy, modify, or distribute the Product, or make any copy, adaptation, transcription, or merge any portion thereof, unless expressly authorized by Gurobi in a separate written agreement."

jamiesnape commented 5 years ago

Ultimately, it would need to be Gurobi. Whether it is you or Russ or MIT who makes the request to Gurobi is a slightly different question. The license on CI is officially assigned to MIT CSAIL with Russ as the administrator.

jwnimmer-tri commented 4 years ago

... just check the license prior to ChooseBestSolver making a decision ...

FYI #13279 has implemented this now.

jamiesnape commented 4 years ago

Note, I don't know of a license that would work for Colab, even if we could redistribute. Pretty much all of Gurobi's licenses end up being attached to either a MAC address or a subnet.

jamiesnape commented 4 years ago

(Gurobi Cloud is their product and it is a different architecture to the C library)

RussTedrake commented 3 years ago

Just noticed this for the first time: https://www.gurobi.com/license-center/for-online-courses/ I wonder if that changes anything?

jamiesnape commented 3 years ago

Difficult to say for certain. Probably not, but it could give you a different avenue to talking to Gurobi. The redistribute part of the license is the problem, and I do not suppose the new license changes that. I doubt it changes anything for Colab, for example.

jwnimmer-tri commented 3 years ago

Another option here might be to write a SolverInterface implementation in pure Python that calls into gurobipy (which is Gurobi's first-party python module). That might avoid the kinds of linker headaches anticipated above, even to the point where we could distribute that binding directly as part of pip install drake. It would not be in pydrake.all, and thus would only conditionally depend in import gurobipy being available.

RussTedrake commented 2 years ago

Just stumbled on this, which might be relevant for the "license on deepnote / colab (if that ever comes back)": https://support.gurobi.com/hc/en-us/articles/4409582394769-Google-Colab-Installation-and-Licensing

jwnimmer-tri commented 1 year ago

@RussTedrake with the uptick in some of the new optimal planning code in Drake and Drake-adjacent projects, it seems like everyone wants Gurobi, so then rebuilds Drake from source, but then loses SNOPT.

I'm thinking about resurrecting this work and trying to reach out to Gurobi people for a redistribution license.

Do you have any input before I start that? (email is fine too)

RussTedrake commented 1 year ago

No updates from me. I agree with your assessment!

jwnimmer-tri commented 1 year ago

To expand upon my proposal above (https://github.com/RobotLocomotion/drake/issues/10804#issuecomment-965861299)...

I'm proposing adding a new class pydrake.solvers.GurobiPySolver that implements SolverInterface in Python.

That would allow Python users of Drake to easily use Gurobi to solve their optimization programs with only a license key, without recompilation. C++ users of Gurobi would still need to rebuild Drake from source.

The ongoing maintenance burden would be that we need to maintain two copies of the gurobi_solver implementation indefinitely. With some careful refactoring for reuse, I hope we could get that down to only ~500 lines of code that's duplicated.

@hongkai-dai could you offer any general feedback or advice about this proposal?

jwnimmer-tri commented 1 year ago

I should also add -- since the program is convex the only Python/C++ calling cost overhead would be while iterating through the costs and constraints. During the solve, the are no nonlinear constraints that would require per-step callbacks crossing the C++/Python GIL boundary.

jwnimmer-tri commented 1 year ago

@RussTedrake @hongkai-dai I would be curious to hear your thoughts about the proposal upthread (see the prior two messages).

My proposal is that we solve this packaging issue for Python users only. Python users would be able to pip install drake gurobipy and then Gurobi would be usable (modulo a license key) for MathematicalProgram. C++ users would still need to rebuild from source. Drake C++ code that calls Solve() would be able to call gurobipy.

After the Python part is working, my proposal is to close this issue, and not attempt any C++ integration for our prepackaged releases.

hongkai-dai commented 1 year ago

Sorry I somehow missed your previous message.

I think implementing GurobiPySolver should work. I have been using gurobi's python interface in my personal project and got relatively familiar with it. I can work on this in my spare time. What is the timeline for adding GurobiPySolver?

jwnimmer-tri commented 1 year ago

I don't have any specific goal in mind for a release date of the feature. I'm sure many users would enjoy having it, but I haven't gotten any recent requests myself. (Russ would probably know more on that front.)

The main purpose of my question was to able to remove the "redistribute libgurobi.so" from my build system task list. If we can agree to give up on C++ support, I can focus my energy elsewhere.

RussTedrake commented 1 year ago

I think it will be an additional support burden (especially for Hongkai). But I agree it seems the best solution.

RussTedrake commented 1 year ago

Do we expect it could result in a significant performance hit?

jwnimmer-tri commented 1 year ago

https://github.com/RobotLocomotion/drake/issues/10804#issuecomment-1523916961 is my current guess re: performance impact. We'll have to see how it turns out, though.

jwnimmer-tri commented 1 year ago

Actually, we may not need to use pygurobi for solving, only for providing the shared library. The beautiful https://github.com/yugr/Implib.so tool lets us do lazy loading of shared libraries the first time they are called, instead of only lazy binding. It seems like Mach-O might already have a similar feature built-in. I will continue to investigate, time permitting.

RussTedrake commented 1 year ago

That would be fantastic!