cysignal, again

[lducas]

I'm having installation issues again. I tried using the --no-cache-dir option for pip, but that was unhelpful :/

> PYTHON=python3 ./bootstrap.sh
Usage:
   ./bootstrap.sh [ -j <#jobs> ]  (uses system's python)
   PYTHON=python2 ./bootstrap.sh  (uses python2)
   PYTHON=python3 ./bootstrap.sh  (uses python3)

Using python version: 3.8.5
Using -j 4 
created virtual environment CPython3.8.5.final.0-64 in 276ms
  creator CPython3Posix(dest=/home/ducas/code/g6k/g6k-env, clear=False, global=False)
  seeder FromAppData(download=False, chardet=latest, idna=latest, progress=latest, retrying=latest, pytoml=latest, webencodings=latest, pip=latest, appdirs=latest, urllib3=latest, six=latest, pep517=latest, CacheControl=latest, setuptools=latest, pyparsing=latest, packaging=latest, certifi=latest, msgpack=latest, contextlib2=latest, distlib=latest, html5lib=latest, pkg_resources=latest, distro=latest, lockfile=latest, wheel=latest, colorama=latest, requests=latest, ipaddr=latest, via=copy, app_data_dir=/home/ducas/.local/share/virtualenv/seed-app-data/v1.0.1.debian)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
ln: failed to create symbolic link './activate': File exists
Collecting pip
  Downloading pip-21.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 6.0 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-21.1
Collecting Cython
  Downloading Cython-0.29.23-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 5.9 MB/s 
Installing collected packages: Cython
Successfully installed Cython-0.29.23
Collecting cysignals
  Downloading cysignals-1.10.3.tar.gz (455 kB)
     |████████████████████████████████| 455 kB 4.4 MB/s 
Requirement already satisfied: Cython>=0.28 in ./g6k-env/lib/python3.8/site-packages (from cysignals) (0.29.23)
Building wheels for collected packages: cysignals
  Building wheel for cysignals (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/ducas/code/g6k/g6k-env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-q3_8841u
       cwd: /tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/
  Complete output (33 lines):
  running bdist_wheel
  running build
  make: 'configure' is up to date.
  checking for a BSD-compatible install... /usr/bin/install -c
  checking whether build environment is sane... yes
  checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
  checking for gawk... gawk
  checking whether make sets $(MAKE)... yes
  checking whether make supports nested variables... yes
  configure: error: source directory already configured; run "make distclean" there first
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py", line 132, in <module>
      setup(
    File "/home/ducas/code/g6k/g6k-env/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/ducas/code/g6k/g6k-env/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py", line 96, in run
      subprocess.check_call(["sh", "configure"])
    File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['sh', 'configure']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for cysignals
  Running setup.py clean for cysignals
Failed to build cysignals
Installing collected packages: cysignals
    Running setup.py install for cysignals ... error
    ERROR: Command errored out with exit status 1:
     command: /home/ducas/code/g6k/g6k-env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-aw3ssj__/install-record.txt --single-version-externally-managed --compile --install-headers /home/ducas/code/g6k/g6k-env/include/site/python3.8/cysignals
         cwd: /tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/
    Complete output (35 lines):
    running install
    running build
    make: 'configure' is up to date.
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
    checking for gawk... gawk
    checking whether make sets $(MAKE)... yes
    checking whether make supports nested variables... yes
    configure: error: source directory already configured; run "make distclean" there first
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py", line 132, in <module>
        setup(
      File "/home/ducas/code/g6k/g6k-env/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/ducas/code/g6k/g6k-env/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
        self.run_command('build')
      File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py", line 96, in run
        subprocess.check_call(["sh", "configure"])
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['sh', 'configure']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/ducas/code/g6k/g6k-env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80ugwfgd/cysignals_1bb7daef886e42e39a32001249e83af4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-aw3ssj__/install-record.txt --single-version-externally-managed --compile --install-headers /home/ducas/code/g6k/g6k-env/include/site/python3.8/cysignals Check the logs for full command output.

[malb]

Looks like configure: error: source directory already configured; run "make distclean" there first is the root cause of things failing.

[lducas]

I'm not sure of where is "there"... For context: I'm just running bootstrap.sh on fresh completely fresh copy of the repo...

[lducas]

Running plain pip3 install --upgrade pip I get a warning pointing to: https://github.com/pypa/pip/issues/9617

No clue if this can be the source of the issue.

[malb]

Have you tried pip install --upgrade Cython (source: https://github.com/fplll/fpylll/issues/96 ?)

That is, change $PIP install Cython to $PIP install -U Cython in bootstrap.sh

[lducas]

I just did. No luck.

[malb]

Another thing to try:

• activate the virtualenv just created, i.e. g6k-env
• git clone https://github.com/sagemath/cysignals
• cd cysignals
• python3 setup.py build
• python3 setup.py install

That is, try to see what happens if you install cysignals by hand.

[lducas]

Same error it seems:

(g6k-env) [13:58:24] ducas@imobill:~/code/g6k/cysignals  
> python3 setup.py build
running build
running build_py
running configure
autoconf
autoheader
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
configure: error: source directory already configured; run "make distclean" there first
Traceback (most recent call last):
  File "setup.py", line 193, in <module>
    setup(
  File "/home/ducas/code/g6k/g6k-env/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 172, in run
    self.run_command('configure')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 142, in run
    subprocess.check_call(['sh', 'configure'] + self.configure_flags)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sh', 'configure']' returned non-zero exit status 1.

[malb]

What happens if you call make distclean and then try again?

[lducas]

No effect. I installed a fresh ubuntu 21.04, and still runs into the same issue.

[malb]

I can't reproduce it. I tried Ubuntu 21.04 first, but that doesn't run inside Docker because of some bug, so I tried 20.04 instead:

apt update
apt upgrade -y
apt install -y python3 python3-dev git build-essential python3-setuptools python3-pip automake autoconf libtool
git clone https://github.com/sagemath/cysignals/
cd cysignals/
python3 setup.py build

This works for me.

[cr-marcstevens]

I can confirm it works on a freshly installed Ubuntu 21.04 Desktop version with the following commands:

sudo apt-get install build-essential autoconf automake libtool pkg-config git
sudo apt-get install libmpfr-dev libgmp-dev python3 python3-dev python3-pip python3-setuptools
python3 -m pip install virtualenv

git clone https://github.com/fplll/g6k
cd g6k
PYTHON=python3 ./bootstrap.sh

[lducas]

Ok, thanks for these useful information. That allowed me to narrow it down: the only thing that survived my re-install was the home folder, including my .bashrc. Removing

alias python=python3
alias pip=pip3

from it solved the issue.

[lducas]

(Feel free to keep commenting, in particular pointing out at why making those aliases was a terrible idea.)