Interface cryptominisat 5

[edd8e884-f507-429a-b577-5d554626c0fe]

This ticket is a compagnon of #22817 which updates cryptominisat package to 5.0.1. Since our current cython interface does not work with cryptominisat 5, we have to rewrite the interface to use the python bindings that are now provided by cryptominisat.

The tarball is at https://github.com/msoos/cryptominisat/archive/5.0.1.tar.gz

(might have to renamed---or downloaded using a browser)

Depends on #22817

CC: @kiwifb

Component: packages: experimental

Keywords: days86, thursdaysbdx, sdl

Author: Thierry Monteil

Branch: c22de1d

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/22818

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0566102 to e382251

[seblabbe]

comment:40

Still, there should be at least one EXAMPLES:: in __init__ for 100% coverage reason.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e382251 to 5236281

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5236281

#22818 : comment 40.

[kiwifb]

comment:42

You should rebase the branch of this ticket on the new branch of #22817.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:43

Replying to @kiwifb:

You should rebase the branch of this ticket on the new branch of #22817.

Sure, i must first compile the changes made on #22817, and test it.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6e93c20	Merge branch 'develop' into package_cryptominisat_5
a8db509	patch to get install_name to be properly installed on OS X. Subtle QA for install by using GNUInstallDirs - may need more love.
fd86b98	Make sure we find sage's zlib rather the system one.
6ed1c73	Merge branch 't/22818/interface_cryptominisat_5' into t/22817/package_cryptominisat_5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5236281 to 6ed1c73

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:45

Done.

[seblabbe]

comment:46

On OSX 10.10, on top of #22999 and with #22818, I get:

$ sage -f cryptominisat
...
[cryptominisat-5.0.1] Successfully installed cryptominisat-5.0.1

Then, I still get:

sage: import pycryptosat
Traceback (most recent call last)
<ipython-input-1-1cca6683c235> in <module>()
----> 1 import pycryptosat

ImportError: dlopen(/Users/slabbe/Applications/sage-git/local/lib/python2.7/site-packages/pycryptosat.so, 2): Library not loaded: libcryptominisat5.5.0.dylib
  Referenced from: /Users/slabbe/Applications/sage-git/local/lib/python2.7/site-packages/pycryptosat.so
  Reason: image not found

I do not know if there is something else I must do to install the new version of the branch? Do some rm in the lib folder to erase previous wrong cryptominisat lib from previous tries? Tell me if so.

Here is the output of previous commands you asked me:

$ find local -name '*cryptominisat*'
local/bin/cryptominisat5_simple
local/include/cryptominisat5
local/include/cryptominisat5/cryptominisat.h
local/include/cryptominisat5/cryptominisat_c.h
local/lib/cmake/cryptominisat5
local/lib/cmake/cryptominisat5/cryptominisat5Config.cmake
local/lib/cmake/cryptominisat5/cryptominisat5Targets-relwithdebinfo.cmake
local/lib/cmake/cryptominisat5/cryptominisat5Targets.cmake
local/lib/libcryptominisat5.5.0.dylib
local/lib/libcryptominisat5.dylib
local/lib/python2.7/site-packages/sage/sat/solvers/cryptominisat
local/lib/python2.7/site-packages/sage/sat/solvers/cryptominisat.py
local/lib/python2.7/site-packages/sage/sat/solvers/cryptominisat.pyc
local/var/lib/sage/installed/cryptominisat-5.0.1

$ find local -name '*cryptosat*'
local/lib/python2.7/site-packages/pycryptosat-5.0.1-py2.7.egg-info
local/lib/python2.7/site-packages/pycryptosat.so

Here is the output of otool as above if this may help:

$ otool -L lib/libcryptominisat5.5.0.dylib
lib/libcryptominisat5.5.0.dylib:
    /Users/slabbe/Applications/sage-git/local/lib/libcryptominisat5.5.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libm4ri-0.0.20140914.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.20.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /Users/slabbe/Applications/sage-git/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

$ otool -L bin/cryptominisat5_simple
bin/cryptominisat5_simple:
    /Users/slabbe/Applications/sage-git/local/lib/libcryptominisat5.5.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
    /Users/slabbe/Applications/sage-git/local/lib/libm4ri-0.0.20140914.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.20.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /Users/slabbe/Applications/sage-git/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

$ otool -L lib/python2.7/site-packages/pycryptosat.so
lib/python2.7/site-packages/pycryptosat.so:
    build/lib.macosx-10.9-x86_64-2.7/pycryptosat.so (compatibility version 0.0.0, current version 0.0.0)
    libcryptominisat5.5.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1152.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.20.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

[kiwifb]

comment:47

lib/python2.7/site-packages/pycryptosat.so is not normal. My install shows

Mirage:sage-build fbissey$ otool -L local/lib/python2.7/site-packages/pycryptosat.so 
local/lib/python2.7/site-packages/pycryptosat.so:
    build/lib.macosx-10.9-x86_64-2.7/pycryptosat.so (compatibility version 0.0.0, current version 0.0.0)
    /Users/fbissey/build/sage-build/local/lib/libcryptominisat5.5.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /Users/fbissey/build/sage-build/local/lib/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1349.8.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)

For some reason you appear to have an old install in the python folder. I didn't have to do any rm but removing that file before re-installing may be something to try.

[seblabbe]

comment:48

I did:

rm local/lib/python2.7/site-packages/pycryptosat.so
sage -f cryptominisat

I get:

$ otool -L local/lib/python2.7/site-packages/pycryptosat.so
local/lib/python2.7/site-packages/pycryptosat.so:
    build/lib.macosx-10.9-x86_64-2.7/pycryptosat.so (compatibility version 0.0.0, current version 0.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libcryptominisat5.5.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1152.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.20.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /Users/slabbe/Applications/sage-git/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

and now:

sage: import pycryptosat
sage: from pycryptosat import Solver
sage: s = Solver()
sage: s.add_clause((1r,1r,1r,1r,-1r))
sage: s.solve()
(True, (None, False))

which is great! Should I sage -f -c cryptominisat now?

[kiwifb]

comment:49

Replying to @seblabbe:

I did:

rm local/lib/python2.7/site-packages/pycryptosat.so
sage -f cryptominisat

and now:

sage: import pycryptosat
sage: from pycryptosat import Solver
sage: s = Solver()
sage: s.add_clause((1r,1r,1r,1r,-1r))
sage: s.solve()
(True, (None, False))

Looks good. I am not sure why the old one stuck around. It must have tricked cmake in being up to date.

Should I sage -f -c cryptominisat now?

????

[seblabbe]

comment:50

Should I sage -f -c cryptominisat now?

????

Indeed, I was confusing with #22999.

Should #22999 be a dependency of #22818, since it is now necessary for some versions of osx?

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:51

It could be a dependency of #218017, which is about building cryptominisat 5. Also, if everything builds fine on supported archs, we could move crymtominisat from experimental to optional (in particular this avoids the need to modify its type by hand when running doctests).

Regarding sage -f -c cryptominisat, there is no self-tests yet, so it will make no difference with sage -f cryptominisat (though it is always a good idea when reviewing a packaging ticket).

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:52

That said, i would really like to have cryptominisat 5 entering Sage 8.0, since i rely on sat solvers for some tutorials, where some people run the 32bit Sage Debian Live USB.

[seblabbe]

comment:53

On osx 10.10 with 8.0.beta7 + #22999 + #22818 + this change:

diff --git a/build/pkgs/cryptominisat/type b/build/pkgs/cryptominisat/type
index 9839eb2..134d9bc 100644
--- a/build/pkgs/cryptominisat/type
+++ b/build/pkgs/cryptominisat/type
@@ -1 +1 @@
-experimental
+optional

the command:

sage -t --long src/sage/sat/solvers/cryptominisat.py src/sage/sat/boolean_polynomials.py src/sage/sat/converters/polybori.py src/sage/sat/solvers/satsolver.pyx src/sage/rings/polynomial/multi_polynomial_sequence.py

gives

Using --optional=4ti2,atlas,cbc,ccache,cryptominisat,latte_int,lidia,lrslib,mpir,pandoc_attributes,pandocfilters,python2,rst2ipynb,sage
Doctesting 5 files.
sage -t --long src/sage/sat/solvers/cryptominisat.py
    [49 tests, 1.16 s]
sage -t --long src/sage/sat/boolean_polynomials.py
    [27 tests, 1.98 s]
sage -t --long src/sage/sat/converters/polybori.py
    [121 tests, 1.21 s]
sage -t --long src/sage/sat/solvers/satsolver.pyx
    [42 tests, 1.20 s]
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py
    [237 tests, 11.70 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

[seblabbe]

Reviewer: Sébastien Labbé

[seblabbe]

comment:54

To me this is good to go and I am ok with changing type to optional.

I did not check that documentation builds fine though (overnight my computer will work on #22999 instead).

[seblabbe]

comment:55

I builded the documentation but I can't find cryptominisat doc in the reference manual:

sage/sat/solvers/cryptominisat should be listed in src/doc/en/reference/sat/index.rst in the following section:

Details on Specific Solvers
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. toctree::
   :maxdepth: 1

   sage/sat/solvers/satsolver
   sage/sat/solvers/dimacs
   sage/sat/solvers/sat_lp
.. optional - cryptominisat
.. sage/sat/solvers/cryptominisat/cryptominisat
.. sage/sat/solvers/cryptominisat/solverconf

In the same file, is the following paragraph still up to date?

Furthermore, Sage provides a C++ interface to the *CryptoMiniSat* [CMS]_
SAT solver which can be used interchangably with DIMACS-based solvers, but also
provides advanced features. For this last solver, the optional CryptoMiniSat
package must be installed, this can be accomplished by typing the following in the
shell::

    sage -i cryptominisat sagelib

In cryptominisat.py, it should be `` instead of ' :

    - 'confl_limit' -- an integer (default: ``None``). Abort after this many
      conflicts. If set to ``None``, never aborts.

Similar problem some lines below (I think this one should break the doc building) :

    - ``filename'' - ...

In input blocks, - should be changed to -- to adhere to the sage convention (several instances):

-        - ``decision`` - accepted for compatibility with other solvers, ignored.
+        - ``decision`` -- accepted for compatibility with other solvers, ignored.

Remove the empty line here:

+            sage: from sage.sat.solvers.cryptominisat import CryptoMiniSat
+            sage: solver = CryptoMiniSat()                                  # optional - cryptominisat
+
+            sage: solver.nvars()                                            # optional - cryptominisat
+            0

Finaly, I would replace the top paragraph:

-This solver relies on Python bindings provided by upstream cryptominisat, and
-replaces the cython interface (written by Martin Albrecht in 2012) that does not
-work with recent versions of cryptominisat anymore.
-
-The ``cryptominisat`` package should be installed on your Sage installation.
-
-AUTHORS:
-
-- Thierry Monteil (2017): first version
+This solver relies on Python bindings provided by upstream cryptominisat.
+
+The ``cryptominisat`` package should be installed on your Sage installation.
+
+AUTHORS:
+
+- Thierry Monteil (2017): replaced the cython interface that does not work 
+   with version >=5.0 of cryptominisat anymore.
+- Martin Albrecht (2012): first version

Finally, no dot in the title:

-CryptoMiniSat Solver.
+CryptoMiniSat Solver

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9593ce6	Merge
c9dd355	#22818 : comment 55.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6ed1c73 to c9dd355

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:58

Thanks for the hints. I applied them quite blindly since i do not have enough RAM to build de doc anymore.

[seblabbe]

comment:59

Great. I managed to build the documentation without errors. I managed to see the Cryptominisat documentation in the reference manual. Every things is okay. Only one problem with the following paragraph which

• misses a :: at the end,
• misses a punctuation before Note that
• misses the end of the last sentence?

        Note that in cryptominisat, the DIMACS standard format is augmented with
        the following extension: having an ``x`` in front of a line makes that
        line an XOR clause i Note that cryptominisat has its own 

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c22de1d

#22818 : comment 59.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from c9dd355 to c22de1d

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:62

Sorry, it was an editing noise, the right sentences ends brefore, see https://www.msoos.org/xor-clauses/

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:64

Thanks for all the review, i am so happy that cryptominisat can now be installed on 32bit machines. BTW, shall we move the package from experimental to optional ?

[seblabbe]

comment:65

I think yes. Maybe in another ticket.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:66

Replying to @seblabbe:

I think yes. Maybe in another ticket.

OK, let us not miss the 8.0 with a new round for that.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:67

This is now #23219.

[vbraun]

Changed branch from u/tmonteil/interface_cryptominisat_5 to c22de1d

[seblabbe]

comment:69

See #23351 for a follow up ticket.

[seblabbe]

Changed commit from c22de1d to none

[edd8e884-f507-429a-b577-5d554626c0fe]

Changed keywords from days86, thursdaysbdx to days86, thursdaysbdx, sdl