search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.34k stars 1.26k forks source link

Redistribute mosek in Drake binary releases #16439

Closed RussTedrake closed 2 years ago

RussTedrake commented 2 years ago

Related to #10804, but it deserves it own issue for tracking.

Problem: Currently no solver that is available in the Drake binary release supports mixed-integer optimization, which in increasingly important. Mosek is also a superior solver relative to many of the open-source solvers we make available for a number of optimization classes.

Proposed solution: Drake binary packages include the mosek binary; users would still have to provide the mosek license file.

This may involve an agreement w/ Mosek.

jwnimmer-tri commented 2 years ago

The Mosek EULA section 8 discusses re-hosting. I only spent a few moments reading it, but it seems like there is not a blanket grant for us to redistribute. It does seem like we would need to get some kind of special permission. Or possibly there is a redistribution grant somewhere else, that I've overlooked.

Or possibly we could have a drake "installer" that downloads and installs Mosek, adjacent to a Drake tarball. We might not be able to add it to our Docker or Pip images, though.

The Mosek wheel (https://pypi.org/project/Mosek/#files) for pip install Mosek has the runtime library embedded in it:

  -rwxr-xr-x    324676  24-Jan-2022  09:09:38  mosek/libcilkrts.so.5
  -rwxr-xr-x  27536576  24-Jan-2022  09:09:38  mosek/libmosek64.so.9.3

For pip install drake, we might be able to reference that directly (instead of redistributing our own copy), as a dependency.

The question would be if it's okay (for our taste) for (say) pip install drake==v0.39.0 to have an unconditional dependency on (say) requirements: mosek >= 9.3, < 9.4, or whether it would need to be somehow optional and opted-in at runtime.

In any case, if we could get permission to redistribute the Mosek runtime library (no need for headers) alongside Drake, that would be the easiest way forward. We could include any notices or documentation that were important, as requested.

jwnimmer-tri commented 2 years ago

@BetsyMcPhail -- Russ is working on the "permission" facet above, and will probably be finished soon. I've added this to "On Deck" in anticipation.

After that's ready, it's over to us to work on the implementation here to make it happen. The plan will be to turn on MOSEK during our Packaging builds (in addition to SNOPT).

I imagine it will be something like this:

RussTedrake commented 2 years ago

FYI. We now have a fully-executed agreement with Mosek allowing us to redistribute the Mosek binaries with the Drake binaries. Hoorah! @BetsyMcPhail -- Let us please move forward on this.

jwnimmer-tri commented 2 years ago

I'll sort this into the work queue in the appropriate place. We need to switch to Focal (and related CI work) before we start this.

BetsyMcPhail commented 2 years ago

While I was making changes to Jenkins, I added the following experimental jobs:

Nothing else has been updated but they are ready for testing when needed

jwnimmer-tri commented 2 years ago

Ensure that when MOSEK is enabled in the build, that the proper files and notices are installed (if not already), ...

I double-checked our license agreement and ensuing discussions.

The runtime files we need are the two shared libraries we've already installed (libmosek, libcilkrts).

The only documentation files we need are just the two:

Those seem to be already installed when MOSEK is enabled, but we should double-check in the new CI packaging builds.

... and that any disclaimers we need on the website appear there now.

I still need to publish the new license contract document before we can turn on MOSEK for the nightlies. I'm working on that.

BetsyMcPhail commented 2 years ago

ensure that the drake-ci scripts notice this and correctly enable both SNOPT and MOSEK in the build and its test filters.

From the new linux-focal-unprovisioned-gcc-bazel-experimental-snopt-mosek-packaging job:

Bazel Command: CC=gcc CXX=g++ bazel run --compilation_mode=opt --config=mosek --config=snopt --test_tag_filters=-gurobi //:install -- /opt/drake

So no changes needed to the CI scripts.

BetsyMcPhail commented 2 years ago

A PR to change all "snopt-packaging" jobs to "snopt-mosek-packaging" has been created. It is waiting and ready to coordinate with the other pieces of this issue.

jwnimmer-tri commented 2 years ago

I still need to publish the new license contract document before we can turn on MOSEK for the nightlies. I'm working on that.

Filed as => #16934. Once that merges, we should be clear to land the PRs that turn on Mosek in our binaries.

BetsyMcPhail commented 2 years ago

Wheel job names in Jenkins have also been updated in the same drake-jenkins-jobs PR.