Installation fails on Ubuntu 14.04

[hiroe-]

Hi, I am trying to install firedrake using python 3.5.1 on Ubuntu 14.04. It fails when installing h5py:

Command "/home/hyamazak/firedrake/bin/python -c "import setuptools, tokenize;__file__='/home/hyamazak/firedrake/src/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /home/hyamazak/firedrake/src/h5py/
Exception information:
Traceback (most recent call last):
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/req/req_install.py", line 845, in install
    install_options, global_options, prefix=prefix)
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/req/req_install.py", line 1007, in install_editable
    show_stdout=False)
  File "/home/hyamazak/firedrake/lib/python3.5/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "/home/hyamazak/firedrake/bin/python -c "import setuptools, tokenize;__file__='/home/hyamazak/firedrake/src/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /home/hyamazak/firedrake/src/h5py/

The entire log file attached. Any advise would be appreciated. firedrake-install.log

[miklos1]

@hiroe-: where is your Python 3.5 installation from?

@dham, @wence-: Is Ubuntu 14.04 officially supported or are we expecting 16.04?

[wence-]

This appears to be this bug: cython/cython#1880. The Cython wheel that has been installed linked against 3.5.3, but Hiroe has 3.5.1, and Python made an ABI-incompatible change in a point release (AAAARGH!).

We could switch to building cython ourselves, or hope they fix it soon? Or pin at 0.26?

@hiroe- if you do:

python -m venv tmp
. tmp/bin/activate
pip install cython==0.26
python -c "from Cython.Build import cythonize"

Do things work?

[hiroe-]

@miklos1 I installed python 3.5.1 using pyenv

@wence-

python -m venv tmp
. tmp/bin/activate
pip install cython==0.26
python -c "from Cython.Build import cythonize"

This gave me the same error at h5py. However when I installed python 3.5.3 and do firedrake-install again, it successfully installed firedrake.

Now I did make alltest and got a lot of

*** Error in `/home/hyamazak/firedrake/bin/python3': free(): invalid pointer: 0x00007f7005297f80 ***
[ella:00927] *** Process received signal ***
[ella:00927] Signal: Aborted (6)
[ella:00927] Signal code:  (-6)
[ella:00927] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f70079c0330]
[ella:00927] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f7006efec37]
[ella:00927] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f7006f02028]
[ella:00927] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x732a4) [0x7f7006f3b2a4]
[ella:00927] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x7f82e) [0x7f7006f4782e]
[ella:00927] [ 5] /home/hyamazak/firedrake/bin/python3() [0x47411f]
[ella:00927] [ 6] /home/hyamazak/firedrake/bin/python3(PyImport_Cleanup+0x2a8) [0x51a3e8]
[ella:00927] [ 7] /home/hyamazak/firedrake/bin/python3() [0x52cfb9]
[ella:00927] [ 8] /home/hyamazak/firedrake/bin/python3(Py_Main+0x5d7) [0x420797]
[ella:00927] [ 9] /home/hyamazak/firedrake/bin/python3(main+0x157) [0x41c677]
[ella:00927] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7006ee9f45]
[ella:00927] [11] /home/hyamazak/firedrake/bin/python3() [0x41c71e]
[ella:00927] *** End of error message ***
Aborted (core dumped)

Any idea what is causing the problem?

[wence-]

I think that is cython/cython#1890. I have pushed a fix for new installs. For the existing build (which is not working) do:

pip uninstall cython
pip install cython==0.26
firedrake-update --rebuild

Thanks.

[hiroe-]

Thanks, the bunch of free() errors went away!

Now make alltest fails only one test:

=================================== FAILURES ===================================
_____________________ test_demo_runs[navier_stokes.py.rst] _____________________

py_file = '/tmp/pytest-of-hyamazak/pytest-463/test_demo_runs_navier_stokes_p0/navier_stokes.py'
env = {'AMMM_SRC': '/home/hyamazak/OpenFOAM/hyamazak-dev/AMMM/src', 'ATMOSFOAM_SRC': '/home/hyamazak/OpenFOAM/hyamazak-dev/A...FOAM/hyamazak-dev/AtmosFOAM-tools/s
rc', 'FIREDRAKE_TSFC_KERNEL_CACHE_DIR': '/home/hyamazak/firedrake/.cache/tsfc', ...}

    def test_demo_runs(py_file, env):
>       subprocess.check_call([sys.executable, py_file], env=env)

/home/hyamazak/firedrake/src/firedrake/tests/demos/test_demos_run.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = (['/home/hyamazak/firedrake/bin/python', '/tmp/pytest-of-hyamazak/pytest-463/test_demo_runs_navier_stokes_p0/navier_stokes.py'],)
kwargs = {'env': {'AMMM_SRC': '/home/hyamazak/OpenFOAM/hyamazak-dev/AMMM/src', 'ATMOSFOAM_SRC': '/home/hyamazak/OpenFOAM/hyamaz...OAM/hyamazak-dev/AtmosFOAM-tools/src', 'FIREDRAKE_TSFC_KERNEL_CACHE_DIR': '/home/hyamazak/firedrake/.cache/tsfc', ...}}
retcode = 1
cmd = ['/home/hyamazak/firedrake/bin/python', '/tmp/pytest-of-hyamazak/pytest-463/test_demo_runs_navier_stokes_p0/navier_stokes.py']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/home/hyamazak/firedrake/bin/python', '/tmp/pytest-of-hyamazak/pytest-463/test_demo_runs_navier_stokes_p0/navier_stokes.py']' returned non-zero exit status 1

/home/hyamazak/.pyenv/versions/3.5.3/lib/python3.5/subprocess.py:271: CalledProcessError

Is it a test on parallel? In that case I don't really care as I don't do parallel runs.

alltest.log

[wence-]

Hmm, that's strange. This test just runs the Navier-Stokes demo with matrix-free preconditioning. Can you paste the output of:

# In the top-level firedrake directory
$ cd demos
$ make
$ cd matrix_free
$ python navier_stokes.py

?

[hiroe-]

It says:

  0 SNES Function norm 1.525817485902e-01 
  1 SNES Function norm 5.055456776782e-03 
  2 SNES Function norm 5.174213856635e-04 
  3 SNES Function norm 8.114643705744e-06 
Traceback (most recent call last):
  File "navier_stokes.py", line 60, in <module>
    "pc_factor_mat_solver_package": "mumps"})
  File "/home/hyamazak/firedrake/src/firedrake/firedrake/solving.py", line 121, in solve
    _solve_varproblem(*args, **kwargs)
  File "/home/hyamazak/firedrake/src/firedrake/firedrake/solving.py", line 169, in _solve_varproblem
    solver.solve()
  File "/home/hyamazak/firedrake/src/firedrake/firedrake/variational_solver.py", line 221, in solve
    solving_utils.check_snes_convergence(self.snes)
  File "/home/hyamazak/firedrake/src/firedrake/firedrake/solving_utils.py", line 231, in check_snes_convergence
    %s""" % (snes.getIterationNumber(), msg))
firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge after 3 nonlinear iterations.
Reason:
   DIVERGED_LINE_SEARCH

Thanks! navier_stokes.log

[wence-]

Hmm, I have no idea what that problem is. Somehow the LU solve fails at the fourth SNES iteration, but that output is identical to what I have in a successful run.

[hiroe-]

I still have the error but I shall go ahead as I passed all the tests in gusto. Thanks for your help and feel free to close the issue.