Upgrade cryptominisat to 5.8.0, fix build on Cygwin

[embray]

This is to upgrade to CryptoMiniSat 5.8.0, released 2020-07-06.

Tarball at: https://github.com/msoos/cryptominisat/archive/refs/tags/5.8.0.tar.gz

Release notes for the versions since 5.6.8 (current in Sage):

== 5.8.0 ==

Massive new release! Many-many improvements:

• Gauss-Jordan elimination is enabled by default
• Target Phases ("Stable Polarities") are used
• CCAnr SLS solver is enabled and is set to work by default
• Hybrid variable branching heuristics
• Many-many speed improvements
• Better DIMACS parsing to work for other systems
• Made to work with new ApproxMC and UniGen

== 5.7.1 ==

• Removing LSIDS, as it was interfering with performance.

== 5.7.0 ==

• Improved parameters
• Hybrid branching strategies
• and much more

Previous upgrade:

• 27319: Upgrade to Cryptominisat 5.6.8 -- merged in Sage 8.9.beta2

Upstream: Fixed upstream, in a later stable release.

CC: @slel @tscrim @jm58660 @embray @jdemeyer @saraedum @sagetrac-tmonteil @videlec @orlitzky @novoselt @kiwifb @dimpase

Component: packages: optional

Keywords: upgrade, cryptominisat

Author: Erik Bray, Thierry Monteil, François Bissey, Matthias Koeppe

Branch: 69033db

Reviewer: Travis Scrimshaw, Julian Rüth, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/25374

[kiwifb]

comment:84

OK, no possible override, it is not visible from the interface. Disabling python on cygwin is the only we'll build this out of the box.

[kiwifb]

comment:85

Is the python build used on cygwin only static? Or is there a dynamic library somewhere?

[tscrim]

comment:86

Replying to @kiwifb:

Is the python build used on cygwin only static? Or is there a dynamic library somewhere?

I have no idea. How can I check this?

[kiwifb]

comment:87

I don't know, I am not a windows person at all. I don't know how cygwin works. I am inspecting the source to figure out whether this particular is ever used. It may be possible to just edit it out.

[tscrim]

comment:88

It seems like it is dynamic:

tscri@DESKTOP-I0BK7J0 ~/sage
$ ./sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/tscri/sage

(sage-sh) tscri@DESKTOP-I0BK7J0:sage$ which python
/home/tscri/sage/local/var/lib/sage/venv-python3.9/bin/python

(sage-sh) tscri@DESKTOP-I0BK7J0:sage$ exit
Exited Sage subshell.

tscri@DESKTOP-I0BK7J0 ~/sage
$ ls -al /home/tscri/sage/local/var/lib/sage/venv-python3.9/bin/python
-rwxr-xr-x 1 tscri tscri 9747 Jun 22 11:05 /home/tscri/sage/local/var/lib/sage/venv-python3.9/bin/python

tscri@DESKTOP-I0BK7J0 ~/sage
$ ldd local/var/lib/sage/venv-python3.9/bin/python
        ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffd1ac00000)
        KERNEL32.DLL => /cygdrive/c/Windows/System32/KERNEL32.DLL (0x7ffd1a0d0000)
        KERNELBASE.dll => /cygdrive/c/Windows/System32/KERNELBASE.dll (0x7ffd180e0000)
        cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)
        libpython3.9.dll => /usr/bin/libpython3.9.dll (0x3fcb60000)
        cygintl-8.dll => /usr/bin/cygintl-8.dll (0x3fe870000)
        cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x3ff490000)
        cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3fe900000)

Here is also the Cygwin Python's install, which seems to be different:

tscri@DESKTOP-I0BK7J0 ~/sage
$ python
Python 3.9.10 (main, Jan 20 2022, 21:37:52)
[GCC 11.2.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

tscri@DESKTOP-I0BK7J0 ~/sage
$ which python
/usr/bin/python

tscri@DESKTOP-I0BK7J0 ~/sage
$ ldd /usr/bin/python
        ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffd1ac00000)
        KERNEL32.DLL => /cygdrive/c/Windows/System32/KERNEL32.DLL (0x7ffd1a0d0000)
        KERNELBASE.dll => /cygdrive/c/Windows/System32/KERNELBASE.dll (0x7ffd180e0000)
        cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000)
        libpython3.9.dll => /usr/bin/libpython3.9.dll (0x3fcb60000)
        cygintl-8.dll => /usr/bin/cygintl-8.dll (0x3fe870000)
        cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x3ff490000)
        cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3fe900000)

[kiwifb]

comment:89

Good, to know. The bits that are blocking cmake are not actually doing anything in practise. So we can edit them out. Are you OK with trying a patch?

[tscrim]

comment:90

Replying to @kiwifb:

Good, to know. The bits that are blocking cmake are not actually doing anything in practise. So we can edit them out. Are you OK with trying a patch?

Yep, just tell me what to do.

[kiwifb]

Attachment: cryptominisat-5.8.0-rm_extra_python_vars.patch.gz

experimental patch for cryptominisat on cygwin

[kiwifb]

comment:91

OK, I have just attached a patch to the ticket. From the sage root create the folder build/pkgs/cryptominisat/patches/ and copy the patch in that folder.

Once done you can proceed with the normal ./sage -i cryptominisat.

[tscrim]

comment:92

With the patch, it built successfully.

[kiwifb]

comment:93

Great, I believe the patch is platform neutral because it only remove parts that are functionally dead. I will update the branch and once we had some retest on other platform we can get this in.

[tscrim]

comment:94

Now are tests in src/sage/sat/solvers/cryptominisat.py expected to pass? They are marked by pycryptosat, and when I run

$ ./sage -tp --optional=sage,pycryptosat src/sage/sat/solvers/cryptominisat.py

I get lots of failures coming down to this:

**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 50, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - pycryptosat
Exception raised:
    Traceback (most recent call last):
      File "sage/misc/lazy_import.pyx", line 253, in sage.misc.lazy_import.LazyImport._get_object (build/cythonized/sage/misc/lazy_import.c:2942)
        self._object = getattr(__import__(self._module, {}, {}, [self._name]), self._name)
    ModuleNotFoundError: No module named 'pycryptosat'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/tscri/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/tscri/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.sat.solvers.cryptominisat.CryptoMiniSat[1]>", line 1, in <module>
        solver = CryptoMiniSat()                                  # optional - pycryptosat
      File "/home/tscri/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        self._solver = Solver(verbose=int(verbosity), confl_limit=int(confl_limit), threads=int(threads))
      File "sage/misc/lazy_import.pyx", line 391, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:4237)
        return self.get_object()(*args, **kwds)
      File "sage/misc/lazy_import.pyx", line 217, in sage.misc.lazy_import.LazyImport.get_object (build/cythonized/sage/misc/lazy_import.c:2619)
        return self._get_object()
      File "sage/misc/lazy_import.pyx", line 256, in sage.misc.lazy_import.LazyImport._get_object (build/cythonized/sage/misc/lazy_import.c:3057)
        raise FeatureNotPresentError(self._feature, reason=f'Importing {self._name} failed: {e}')
    sage.features.FeatureNotPresentError: pycryptosat is not available.
    Importing Solver failed: No module named 'pycryptosat'
    No equivalent system packages for cygwin are known to Sage.
    To install pycryptosat using the Sage package manager, you can try to run:
      !sage -i cryptominisat
    No equivalent system packages for pip are known to Sage.

[kiwifb]

comment:95

Ok it installed but obviously it doesn't work. There should be a file with a name starting with pycryptominisat inside the site-packages folder of python. Can you see it? and if so what is the exact name?

[kiwifb]

Changed branch from u/dimpase/packages/cryptominisat-5.8.0 to u/fbissey/cryptominisat-5.8.0

[kiwifb]

New commits:

2c68a50	Merge branch 'develop' into cryptominisat-5.8.0
ae01a89	Add patch to cryptominisat removing useless bits preventing compilation on cygwin.

[kiwifb]

Changed commit from ca6d123 to ae01a89

[tscrim]

comment:97

Because I was stupid and didn't realize that was a separate package because I probably mistyped the name when doing ./sage -i pycryptosat. Although I am back with a very familiar error:

[pycryptosat-5.8.0] -- Python CFLAGS:  '-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall'
[pycryptosat-5.8.0] -- Python LDFLAGS: '-lcrypt -lintl -ldl'
[pycryptosat-5.8.0] -- Python LINKFORSHARED flags: ''
[pycryptosat-5.8.0] -- Python module installation prefix: --prefix=/home/tscri/sage/local/var/lib/sage/venv-python3.9
[pycryptosat-5.8.0] -- Configuring incomplete, errors occurred!
[pycryptosat-5.8.0] See also "/home/tscri/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/pycryptosat-5.8.0/src/CMakeFiles/CMakeOutput.log".
[pycryptosat-5.8.0] See also "/home/tscri/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/pycryptosat-5.8.0/src/CMakeFiles/CMakeError.log".
[pycryptosat-5.8.0] ****************************************************************************************************************************************************
[pycryptosat-5.8.0] Error configuring pycryptosat-5.8.0 with cmake
[pycryptosat-5.8.0] See the file
[pycryptosat-5.8.0]     /home/tscri/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/pycryptosat-5.8.0/src/CMakeFiles/CMakeOutput.log
[pycryptosat-5.8.0] for details.
[pycryptosat-5.8.0] ****************************************************************************************************************************************************

[kiwifb]

comment:98

Go figure, it needs patching too since it is the same package. I am going to update the branch.

[tscrim]

comment:99

I am now trying with the same patch copied over. This seems to be working...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ae01a89 to fc04bd5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

fc04bd5

link patches from cryptominisat to pycryptosat since they are the same package and need the same fix.

[tscrim]

comment:101

That worked and the tests in that file pass.

[kiwifb]

comment:102

Cool and now that the branch is updated we should make sure ./sage -i pycryptosat works properly from this branch.

[kiwifb]

comment:103

And I should upstream the thing too, fixing this was just a matter of removing dead branches :)

[tscrim]

comment:104

I just did a force rebuild of ./sage -i -f pycryptosat with the new branch and it is working.

[kiwifb]

Changed author from Erik Bray, Thierry Monteil to Erik Bray, Thierry Monteil, François Bissey

[kiwifb]

Changed reviewer from Julian Rüth, https://github.com/sagemath/sagetrac-mirror/actions/runs/1704383865 to Travis Scrimshaw, Julian Rüth, https://github.com/sagemath/sagetrac-mirror/actions/runs/1704383865

[kiwifb]

Changed dependencies from #33162, #33183 to none

[kiwifb]

comment:105

I have added myself to authors, does anyone need to add themselves? Dima?

Next we need some reviewers to put their hands up. @tscrim just did cygwin so I added him there.

[kiwifb]

Changed work issues from see #33162 to none

[tscrim]

comment:106

The force rebuild finished and the tests in files that mention pycryptosat and cryptominisat passed on Cygwin.

[mkoeppe]

comment:107

We should also add https://github.com/msoos/cryptominisat/pull/679 as a patch so that the next setuptools upgrade (#33866) does not break cryptominisat

[kiwifb]

comment:108

Very sensible. I'll add it to my branch in the next few hours if no one else does it.

[mkoeppe]

comment:109

On macOS, I also get:

[cryptominisat-5.8.0] cd /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/cryptominisat-5.8.0/src/pycryptosat && /Users/mkoeppe/s/sage/sage-rebasing/local/bin/cmake -E env CFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk\ -g\ -O2' /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/bin/python3 setup.py build_ext --inplace --rpath ../lib
[cryptominisat-5.8.0] [100%] Built target cryptominisat5-bin
[cryptominisat-5.8.0] clang: warning: no such sysroot directory: '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -g -O2' [-Wmissing-sysroot]
[cryptominisat-5.8.0] In file included from /Users/mkoeppe/s/sage/sage-rebasing/local/var/tmp/sage/build/cryptominisat-5.8.0/src/pycryptosat/src/pycryptosat.cpp:27:
[cryptominisat-5.8.0] In file included from /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
[cryptominisat-5.8.0] /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/stdio.h:107:15: fatal error: 'stdio.h' file not found

[mkoeppe]

comment:110

This appears to come from https://github.com/msoos/cryptominisat/blob/master/python/CMakeLists.txt#L85

[mkoeppe]

comment:111

I don't see a fix for this in https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cryptominisat.rb

[mkoeppe]

comment:112

cryptominisat master already contains a workaround for likely the same issue of misquoting: https://github.com/msoos/cryptominisat/pull/621/files

[mkoeppe]

comment:113

This whole broken -isysroot thing added in https://github.com/msoos/cryptominisat/commit/8e5c38b8f4099bf558ef1a18b2a8b893460e9943 should probably be removed upstream. But the path of least resistance for us would be to apply https://github.com/msoos/cryptominisat/pull/621 as a patch and to fake NIX_CC.

[mkoeppe]

Changed branch from u/fbissey/cryptominisat-5.8.0 to u/mkoeppe/cryptominisat-5.8.0

[mkoeppe]

Changed commit from fc04bd5 to 6dbbe94

[mkoeppe]

comment:115

Next error:

[cryptominisat-5.8.0] [100%] Built target cryptominisat5-bin
[cryptominisat-5.8.0] In file included from src/pycryptosat.cpp:32:
[cryptominisat-5.8.0] In file included from ../cmsat5-src/cryptominisat5/cryptominisat.h:35:
[cryptominisat-5.8.0] ../cmsat5-src/cryptominisat5/solvertypesmini.h:33:1: error: unknown type name 'constexpr'
[cryptominisat-5.8.0] constexpr uint32_t var_Undef(0xffffffffU >> 4);

---

New commits:

376e3fc	build/pkgs/cryptominisat: Work around for macOS CFLAGS misquoting
6dbbe94	build/pkgs/cryptominisat/patches/679.patch: New

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

bacff4e

build/pkgs/cryptominisat/spkg-install.in: Do not build the Python interface here

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6dbbe94 to bacff4e

[mkoeppe]

comment:117

Now it builds at least cryptominisat for me.

[kiwifb]

comment:118

It should work for pycryptomini provided you add the NIX_CC=fake thing to its spkg-install.in since I made its patches folder a link to the cryptominisat one, so patches would match.

[mkoeppe]

comment:119

pycryptosat currently fails for me with the constexpr problem (same as comment:115). This looks like it is the same as https://github.com/msoos/cryptominisat/issues/684

[mkoeppe]

comment:120

Homebrew fixes it like this: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cryptominisat.rb#L39

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bacff4e to 62e4a96

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

62e4a96

build/pkgs/pycryptosat/spkg-install.in: Use -std=gnu++11

[mkoeppe]

comment:122

Now this builds for me. Strangely I didn't need the NIX_CC workaround. Looks like this broken stuff is only used when the Python module build is done from within the cmake build.