Install of python3.5-jupyter-1.0.0 fails on mac

[michaelmesser]

Issue description

Steps to reproduce

$ nix-env -i python3.5-jupyter-1.0.0
...
/nix/store/pcbrnz1waa12djyk34dpgxs9vqaysr1g-gnused-4.2.2/bin/sed \
        -e 's|[@]PYTHON[@]|/nix/store/52v8vzjgbkz32pbdl4bmdxs35261ly55-python3-3.5.2/bin/python3|g' \
        -e 's|[@]G_TEST_SRCDIR[@]|/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4|g' \
        -e 's|[@]G_TEST_BUILDDIR[@]|/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4|g' \
        test/TestSuitePythonService.service.in > test/TestSuitePythonService.service
/nix/store/pcbrnz1waa12djyk34dpgxs9vqaysr1g-gnused-4.2.2/bin/sed \
        -e 's|[@]PYTHON[@]|/nix/store/52v8vzjgbkz32pbdl4bmdxs35261ly55-python3-3.5.2/bin/python3|g' \
        -e 's|[@]G_TEST_SRCDIR[@]|/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4|g' \
        -e 's|[@]G_TEST_BUILDDIR[@]|/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4|g' \
        test/tmp-session-bus.conf.in > test/tmp-session-bus.conf
make[1]: Leaving directory '/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4'
running tests
check flags: SHELL=/nix/store/716mpwxqg1n49h34ayc8axx591sw88cm-bash-4.3-p42/bin/bash
make  check-TESTS check-local
make[1]: Entering directory '/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4'
make[2]: Entering directory '/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4'
SKIP: test/test-client.py
SKIP: test/test-p2p.py
SKIP: test/test-signals.py
PASS: test/test-standalone.py
PASS: test/test-unusable-main-loop.py
PASS: test/test-exception-py3.py
SKIP: test/run-test.sh
FAIL: test/test-import-repeatedly
=========================================
   dbus-python 1.2.4: ./test-suite.log
=========================================

# TOTAL: 8
# PASS:  3
# SKIP:  4
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

SKIP: test/test-client
======================

SKIP test/test-client.py (exit status: 77)

SKIP: test/test-p2p
===================

SKIP test/test-p2p.py (exit status: 77)

SKIP: test/test-signals
=======================

SKIP test/test-signals.py (exit status: 77)

SKIP: test/run-test
===================

DBUS_TOP_SRCDIR=/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4
DBUS_TOP_BUILDDIR=/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4
PYTHONPATH=/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4:/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4/test:/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4/.libs:/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4/test/.libs
PYTHON=/nix/store/52v8vzjgbkz32pbdl4bmdxs35261ly55-python3-3.5.2/bin/python3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'gi'
could not import python-gi
SKIP test/run-test.sh (exit status: 77)

FAIL: test/test-import-repeatedly
=================================

FAIL test/test-import-repeatedly (exit status: 137)

============================================================================
Testsuite summary for dbus-python 1.2.4
============================================================================
# TOTAL: 8
# PASS:  3
# SKIP:  4
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to http://bugs.freedesktop.org/enter_bug.cgi?product=dbus&component=python
============================================================================
make[2]: *** [Makefile:1689: test-suite.log] Error 1
make[2]: Leaving directory '/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4'
make[1]: *** [Makefile:1797: check-TESTS] Error 2
make[1]: Leaving directory '/private/var/folders/_m/1h8dd_fd3dx3qvtq87_c4q480000gn/T/nix-build-dbus-python-1.2.4.drv-0/dbus-python-1.2.4'
make: *** [Makefile:2017: check-am] Error 2
builder for ‘/nix/store/91rrnljlbdrw9kgvh1c5vfsk10kys7rk-dbus-python-1.2.4.drv’ failed with exit code 2
building path(s) ‘/nix/store/cgsm9yga4fvhzxnqga7igxmpvx0c0cs9-entrypoints-0.2.1-py2.py3-none-any.whl’
cannot build derivation ‘/nix/store/wqj5rr4cndlq1sndkh27nac98lv7wj76-python3.5-PyQt-x11-gpl-4.11.3.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/6vxnsd1c6js5jw18j6iwpk9wry9pydid-python3.5-qtconsole-4.2.1.drv’: 1 dependencies couldn't be built
killing process 18092
cannot build derivation ‘/nix/store/4nj5sjq9bx2hvgnibj3lg1m6g77rjnd5-python3.5-jupyter-1.0.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/4nj5sjq9bx2hvgnibj3lg1m6g77rjnd5-python3.5-jupyter-1.0.0.drv’ failed

Technical details

• System: OS X 10.11.6
• Nix version: 1.11.2
• Nixpkgs version: 16.09pre88706.b8e215f

[knedlsepp]

I haven't cleaned it up nicely for a PR yet, but I think these changes could get us closer to a working jupyter: There seem to be two problems at least:

• Failing test in dbus because of some missing dependency
• pyqt is not building correctly

--- a/pkgs/development/python-modules/dbus/default.nix
+++ b/pkgs/development/python-modules/dbus/default.nix
@@ -17,7 +17,7 @@ if isPyPy then throw "dbus-python not supported for interpreter ${python.executa
     # It seems not to retain the dependency anyway.
     ++ stdenv.lib.optional (! python ? modules) ncurses;

-  doCheck = true;
+  doCheck = false;

pyqt:

+    ${stdenv.lib.optionalString stdenv.isDarwin ''
+      export QMAKESPEC="unsupported/macx-clang-libc++" # OS X target after bootstrapping \
+    ''} \

     substituteInPlace configure.py \
       --replace 'install_dir=pydbusmoddir' "install_dir='$out/lib/${python.libPrefix}/site-packages/dbus/mainloop'" \
+      --replace "qt_macx_spec = 'macx-g++'" "qt_macx_spec = 'unsupported/macx-clang-libc++'" # for bootstrapping

--- a/pkgs/development/python-modules/sip/4.16.nix
+++ b/pkgs/development/python-modules/sip/4.16.nix
@@ -11,7 +11,8 @@ if isPyPy then throw "sip not supported for interpreter ${python.executable}" el
   configurePhase = ''
     ${python.executable} ./configure.py \
       -d $out/lib/${python.libPrefix}/site-packages \
-      -b $out/bin -e $out/include
+      -b $out/bin -e $out/include \
+      --deployment-target 10.6

[knedlsepp]

This is what works for me: https://github.com/knedlsepp/nixpkgs/tree/fix-python3.5-jupyter-1.0.0 However, I don't know how to fix dbus properly. Why the ImportError: No module named 'gi' does not exist on Linux is unclear to me. There is also nothing special on homebrew: https://github.com/Homebrew/homebrew-python/blob/master/python-dbus.rb

[FRidh]

I might have causes the issue with dbus in 5a501bd828b19e9cef4589b3b9dc2852401a2882.

Note that

$ nix-shell -p python pythonPackages.pygobject3 -I nixpkgs=. --run "python -c 'import gi'"
$ nix-shell -p python pythonPackages.pygobject  -I nixpkgs=. --run "python -c 'import gi'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named gi

[FRidh]

so I guess adding pygobject3 to the buildInputs of dbus-python will fix that.

[knedlsepp]

@FRidh Adding ++ stdenv.lib.optionals doCheck [ dbus_tools pygobject3 ] solves the problem, but how come this is only a problem on darwin?

[FRidh]

If I recall correctly the test was skipped op NixOS, but I didn't look at it any further. I've added pygobject3 in https://github.com/NixOS/nixpkgs/commit/3de68a39791df7d518ee59fcc2b9e6576571d2a4

[knedlsepp]

Hm. With the new upstream/master I now get the following for: nix-env -f $NIXPKGS -iA python35Packages.dbus-python

checking for python version... (cached) 3.5
checking for python platform... (cached) darwin
checking for python script directory... (cached) ${prefix}/lib/python3.5/site-packages
checking for python extension module directory... (cached) ${exec_prefix}/lib/python3.5/site-packages
checking for headers required to compile python extensions... ./configure: line 13564: which: command not found
found
checking for libraries required to embed python... ./configure: line 13597: which: command not found
no
configure: error: Python libs not found. Windows and Darwin require Python modules to be explicitly linked to libpython.
builder for ‘/nix/store/w7ym9zy0p4nrzjikbsr42anica4j28di-pygobject-3.20.0.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/xdqjpc12v7qq2qhvp9yp731drhp9wj87-dbus-python-1.2.4.drv’: 1 dependencies couldn't be built

Building pygobject3 fails.

[FRidh]

That seems to be a bit bigger issue. How about adding which to the buildInputs in pkgs/development/python-modules/pygobject/3.nix.

[knedlsepp]

No change in the libpython-regard:

checking for python version... (cached) 3.5
checking for python platform... (cached) darwin
checking for python script directory... (cached) ${prefix}/lib/python3.5/site-packages
checking for python extension module directory... (cached) ${exec_prefix}/lib/python3.5/site-packages
checking for headers required to compile python extensions... found
checking for libraries required to embed python... no
configure: error: Python libs not found. Windows and Darwin require Python modules to be explicitly linked to libpython.
builder for ‘/nix/store/9x99c31cvc2j3ql83rl9pgscj5ihrk0g-pygobject-3.20.0.drv’ failed with exit code 1

[knedlsepp]

I did inspect the config.log file. It seems the problem stems from the following: The configurePhase calls: python3-config --ldflags, which returns: '-lpython3.5m -ldl -lncurses -framework CoreFoundation -Wl,-stack_size,1000000 -framework CoreFoundation' Then it tries to link a program with those libraries and fails in finding ncurses:

configure:13593: checking for libraries required to embed python
configure:13629: clang -o conftest -g -O2   conftest.c  -lpython3.5m -ldl -lncurses -framework CoreFoundation -Wl,-stack_size,1000000 -framework CoreFoundation >&5
ld: library not found for -lncurses
clang-3.7: error: linker command failed with exit code 1 (use -v to see invocation)

So should ncurses be a propagatedBuildInput of python? Or should we just patch the configure.ac to trust us what we are doing? How do you usually solve stuff in nix where an installer says: "I want to have all the dependencies of one of my dependencies!"

[FRidh]

There are plans on including curses with python but this hasn't happened yet. Anyway, you're using python35 which isn't stripped of curses.

The problem is that it wants to link against ncurses. Obviously it cannot find it since it wasn't added as buildInput. Therefore, how about adding ncurses to buildInputs of pygobject3?

[knedlsepp]

Adding both which and ncurses to pygobject3 leads to a working configurePhase. However the buildPhase fails with:

test -L libregress.dylib || ln -s .libs/libregress.dylib libregress.dylib
  CC       testhelper_la-testhelpermodule.lo
  CC       testhelper_la-test-floating.lo
  CC       testhelper_la-test-thread.lo
  CC       testhelper_la-test-unknown.lo
  GEN      testhelper.la
clang-3.7: warning: argument unused during compilation: '-pthread'
ld: -stack_size option can only be used when linking a main executable

This is again because of the python3-config --ldflags:

-lpython3.5m -ldl -lncurses -framework CoreFoundation -Wl,-stack_size,1000000 -framework CoreFoundation

It seems usually python is not configured with "-Wl,-stack_size,1000000" and so the pygobject builder chokes once again.

The 2.7 version does not have this either:

$ python2.7-config --ldflags
-lpython2.7 -ldl -framework CoreFoundation

$ python3.5-config --ldflags
-lpython3.5m -ldl -lncurses -framework CoreFoundation -Wl,-stack_size,1000000 -framework CoreFoundatio

And as a reference the homebrew version of python3.5:

$ python3.5-config --ldflags
-L/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/config-3.5m -lpython3.5m -ldl -framework CoreFoundation

[FRidh]

stack_size isn't set on NixOS. For darwin the following are added to the buildInputs of Python [ CF configd ]. Maybe these set the stack size? @copumpkin , any idea?

[FRidh]

It seems as Python upstream sets the stack size for Darwin http://bugs.python.org/issue14184.

[knedlsepp]

I guess for darwin builds we should peek at homebrew: I think the following patch replaces the stack_size line: LINKFORSHARED="-Wl,-stack_size,1000000 $LINKFORSHARED" https://github.com/Homebrew/homebrew-core/blob/master/Formula/python3.rb#L219

This should get rid of the stack_size stuff:

  prePatch = optionalString stdenv.isDarwin ''
    substituteInPlace configure --replace '`/usr/bin/arch`' '"i386"'
    substituteInPlace configure --replace '-Wl,-stack_size,1000000' ' '
  '';

I will test if this change makes it easier to get the pygobject3 to work. Of course this could break other stuff... (Testing this on: https://github.com/knedlsepp/nixpkgs/commits/fix-python3.5-jupyter-1.0.0)

[knedlsepp]

After rebasing on upstream/master I suddenly get the following problem:

knedlbook:tmp sepp$ nix-env -f $NIXPKGS -iA python35
installing ‘python3-3.5.2’
these derivations will be built:
  /nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv
these paths will be fetched (14.52 MiB download, 14.52 MiB unpacked):
  /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz
fetching path ‘/nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz’...

*** Downloading ‘https://cache.nixos.org/nar/03wvgy4jki2d07slfkv9ixbpa9fwr55nx3p1wxbx6l56n4ql7vf9.nar.xz’ to ‘/nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14.5M  100 14.5M    0     0  2044k      0  0:00:07  0:00:07 --:--:-- 2467k

building path(s) ‘/nix/store/p5cdvfq2qzavaaijlm2p3qwbqyvhs9l8-python3-3.5.2’
unpacking sources
unpacking source archive /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz
dyld: lazy symbol binding failed: Symbol not found: _mkdirat
  Referenced from: /nix/store/iigawjrnbhq0l3nn9k38yybija3fl3f0-gnutar-1.29/bin/tar
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _mkdirat
  Referenced from: /nix/store/iigawjrnbhq0l3nn9k38yybija3fl3f0-gnutar-1.29/bin/tar
  Expected in: /usr/lib/libSystem.B.dylib

/nix/store/v5mjlz5xpcvr8ks8a3yyh5yjvb4xvbia-stdenv-darwin/setup: line 531: 73739 Broken pipe: 13         xz -d < "$fn"
     73740 Trace/BPT trap: 5       | tar xf -
do not know how to unpack source archive /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz
builder for ‘/nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv’ failed with exit code 1
error: build of ‘/nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv’ failed

I don't think this should be related to the changes made by me, but now I can't test if it builds properly.

[FRidh]

maybe this specific error is due to hardening changes? @globin

[copumpkin]

@knedlsepp Are you on 10.9? I think I got rid of some compatibility shims for 10.9 recently that would cause issues like that, because they were a pain to maintain and I figured most Mac users would have moved on by now

On Aug 29, 2016, at 13:58, Josef Kemetmüller notifications@github.com wrote:

After rebasing on upstream/master I suddenly get the following problem:

knedlbook:tmp sepp$ nix-env -f $NIXPKGS -iA python35 installing ‘python3-3.5.2’ these derivations will be built: /nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv these paths will be fetched (14.52 MiB download, 14.52 MiB unpacked): /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz fetching path ‘/nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz’...

*\ Downloading ‘https://cache.nixos.org/nar/03wvgy4jki2d07slfkv9ixbpa9fwr55nx3p1wxbx6l56n4ql7vf9.nar.xz’ to ‘/nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz’... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14.5M 100 14.5M 0 0 2044k 0 0:00:07 0:00:07 --:--:-- 2467k

building path(s) ‘/nix/store/p5cdvfq2qzavaaijlm2p3qwbqyvhs9l8-python3-3.5.2’ unpacking sources unpacking source archive /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz dyld: lazy symbol binding failed: Symbol not found: _mkdirat Referenced from: /nix/store/iigawjrnbhq0l3nn9k38yybija3fl3f0-gnutar-1.29/bin/tar Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _mkdirat Referenced from: /nix/store/iigawjrnbhq0l3nn9k38yybija3fl3f0-gnutar-1.29/bin/tar Expected in: /usr/lib/libSystem.B.dylib

/nix/store/v5mjlz5xpcvr8ks8a3yyh5yjvb4xvbia-stdenv-darwin/setup: line 531: 73739 Broken pipe: 13 xz -d < "$fn" 73740 Trace/BPT trap: 5 | tar xf - do not know how to unpack source archive /nix/store/xbgpq4ph7r3vkqm6ws3rq64ij6yg9hkm-Python-3.5.2.tar.xz builder for ‘/nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv’ failed with exit code 1 error: build of ‘/nix/store/w5lf06s47chz1wrh028x99qgk9abb10g-python3-3.5.2.drv’ failed I don't think this should be related to the changes made by me, but so I can't test if it builds probably.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[knedlsepp]

Yes, I'm on 10.9.5. My commits used to work when based on master from a week ago. Based on yesterdays master I had this strange problem with the lazy symbol binding. Today I get

Undefined symbols for architecture x86_64:
  "_voucher_mach_msg_set", referenced from:
      _configopen in configUser.o
      _configlist in configUser.o
      _configadd in configUser.o
      _configget in configUser.o
      _configset in configUser.o
      _configremove in configUser.o
      _configadd_s in configUser.o
      ...
ld: symbol(s) not found for architecture x86_64
clang-3.7: error: linker command failed with exit code 1 (use -v to see invocation)
builder for ‘/nix/store/l0g6ar3m6f0s6yc8kgiy07977c2gjiyi-configd-osx-10.8.5.drv’ failed with exit code 1

on trying to build my fix-python3.5-jupyter-1.0.0 branch.

[copumpkin]

@knedlsepp I'd open a new ticket that 10.9 isn't supported anymore, to see if there's enough interest in reviving support for it. The commit that broke you is probably https://github.com/NixOS/nixpkgs/commit/948b7f23bbc42a87e66a473c5c4543cfa5a03da7, and note the commit message. Basically I don't really know how to handle something like this: I get almost no help on maintaining the stdenv (partially because nobody knows how to and it's not very documented) but with very limited time, I don't want to keep maintaining support for operating system versions I can't test and that are pretty unlikely to be used (Mac users are generally quick adopters of new OS versions). I'm not saying that you have to step up and help fix the stdenv but if you can open a ticket so we can figure out how many people are affected by us dropping 10.9 support, that'll help me prioritize my efforts. Of course, you stepping up and fixing it yourself would be even better 😄

[knedlsepp]

@copumpkin: I personally avoid upgrading my mac's OS as long as I can, as I always felt that upgrading made my machine more sluggish with every new release. Also rolling back is a pain. I guess in this light I will probably consider an upgrade, as I want to keep using (and contributing to) nix on darwin. I would love to help fixing problems with the stdenv, but as of now this is not within my skills or knowledge about the inner workings of nix or the OS itself. I might go down deeper into the rabbit hole, but this will probably take some time.

[knedlsepp]

Now that https://github.com/NixOS/nixpkgs/pull/18025 is merged, nix-channel --update && nix-env -i python3.5-jupyter-1.0.0 works for me on OS 10.11, so I think this can be closed. @2426021684: Could you try and confirm?

[michaelmesser]

@knedlsepp I can confirm that it works