Ubuntu - Problems with --single-version-externally-managed building Python-Six

[cncpadawan]

Hello!

I am trying to build up a Linux build environment, and am wanting to use the Cura build environment, but in Ubuntu 17.04 and 16.10, I run into the following error:

`[ 12%] Performing install step for 'PythonSix' usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help

error: option --single-version-externally-managed not recognized CMakeFiles/PythonSix.dir/build.make:73: recipe for target 'PythonSix-prefix/src/PythonSix-stamp/PythonSix-install' failed make[2]: [PythonSix-prefix/src/PythonSix-stamp/PythonSix-install] Error 1 CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/PythonSix.dir/all' failed make[1]: [CMakeFiles/PythonSix.dir/all] Error 2 Makefile:83: recipe for target 'all' failed ` For kicks I removed that option for Python-Six, and it built fine, but hung up on the next package (cx_freeze). The files seem to be using the newly compiled Python.

Everything I am finding online points to setuptools being outdated. I downloaded setuptools online, and used the compiled python3 binary to install it, and it was successful with the installation and is now continuing the build of the environment.

Long story short.. something appears to be wrong with setuptools, or it isn't getting installed.

[awhiemstra]

I have noticed that in some cases, setuptools does not install correctly after building Python. I would need a more detailed build log to figure out what is going wrong, however.

[cncpadawan]

Ok.. I can get that for you, just let me know what info you'd like. I have several other projects I had to download the source for and install in a similar fashion, and still not complete yet.

[awhiemstra]

Well primarily I would like to see the Python build output. I suspect it partially fails and then setuptools does not get installed properly.

[cncpadawan]

Hmmm, ok... I'll see about grabbing it all for you. I am rebuilding the environment again... while I was able to successfully build everything, the AppImage would not work (exception right away), like the issue I am having on Mac OSX except it doesn't work as an AppImage or outside of it. Thinking perhaps it has to do with my build environment, so... I'll see what I can do to capture the output.

[cncpadawan]

I have rebuilt the environment, with a difference of NOT running sudo to do the build. Everything built ok, no errors, until I tried building cura-build. It couldn't find PyQT5. I then tried downloading the package that is referenced in cmake and installing it using the parameters in the pyqt.cmake file and it errored out complaining about the license mismatch between it and Qt that is installed in the environment. I then pulled the latest version of PyQT5 down (5.8.2), and it worked just fine with the parameters and no license complaints.

It is currently building to install into the environment, so I'll see how cura-build responds after it is completed.

[cncpadawan]

It seems that Savitar didn't get built again either. I manually built it, installed it using the parameters in the cmake file for the project, and it installed correctly. Cura appears to have built, the AppImage is being built, and then when I try running it, it fails because there is no module named Arcus.

Tried building it following the same cmake parameters.. tried rebuilding using cura-build... same error, no module named Arcus. Searched for the Arcus library, there isn't one in the package directory structure.

Also, I noticed when running (or attempting to run) the AppImage, it is referencing paths in the build environment for things it is looking for - is that normal? Thanks for your help!

[cncpadawan]

Any thoughts on this one?

[Ghostkeeper]

I don't know about the rest, but as far as I understood the AppImage should only have relative references to these things. It gets unpacked in a virtual drive, so having a relative path to this virtual drive would then link to other libraries in the drive rather than your system-libraries or your build environment.

[eloipr]

I get the same error when trying to build cura-build-enviroment on ubuntu 16.04.

Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS
[ 12%] Completed 'Python'
[ 12%] Built target Python
[ 12%] Performing install step for 'PythonSix'
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --single-version-externally-managed not recognized
CMakeFiles/PythonSix.dir/build.make:73: recipe for target 'PythonSix-prefix/src/PythonSix-stamp/PythonSix-install' failed
make[2]: *** [PythonSix-prefix/src/PythonSix-stamp/PythonSix-install] Error 1
CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/PythonSix.dir/all' failed
make[1]: *** [CMakeFiles/PythonSix.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

When building python I get the next warnings:

[ 11%] Performing build step for 'Python'
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c: In function ‘classify_argument’:
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:224:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  for (i = 0; i < words; i++)
                ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:245:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
      for (i = 0; i < num; i++)
                    ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:264:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
      for (i = 1; i < words; i++)
                    ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:270:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  for (i = 0; i < words; i++)
                ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c: In function ‘examine_argument’:
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:323:17: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (i = 0; i < n; ++i)
                 ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c: In function ‘ffi_call’:
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:484:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    for (j = 0; j < n; j++, a += 8, size -= 8)
                  ^
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c: In function ‘ffi_closure_unix64_inner’:
/home/epardo/Documents/Projects/Cura2/cura-build-environment/build/Python-prefix/src/Python/Modules/_ctypes/libffi/src/x86/ffi64.c:659:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    for (j = 0; j < n; j++, a += 8)
                  ^

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _lzma              
_sqlite3              _ssl                  _tkinter           
readline                                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

[ 11%] Performing install step for 'Python'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _lzma              
_sqlite3              _ssl                  _tkinter           
readline                                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

...........

running install
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _lzma              
_sqlite3              _ssl                  _tkinter           
readline                                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

[eloipr]

Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS is the reason why pythonsix cannot be build. Installing openssl and libssl-dev solved the error. sudo apt-get install openssl libssl-dev

[Ghostkeeper]

According to the manifest, openssl should already be installed though.

[LipuFei]

@awhiemstra I am investigating some problems on Fedora 27 and I have also encountered a few issues with setuptools and numpy:

• setuptools is not installed, so I had to add a separate project for it to install from source
• The numpy version 1.11.1 seems to be based on glibc <2.26. On Fedora 27, glibc is 2.26, which has a xlocale.h header file removed, so compiling numpy 1.11.1 always fails on that xlocale.h file. Upgrading it to 1.13.3 seems to work for me.

I would suggest to add a project to install setuptools and upgrade numpy, but I'm not sure if the new numpy will break anything. Any thought?

[asyd]

Same issue here, on a debian:stable container (after tried on my debian/sid). The _ssl module doesn't build, probably because the libssl version is to high regarding what Python 3.5 expect.

dpkg -l libssl-dev
ii  libssl-dev:amd64           1.1.0f-3+deb9u1

According of Python3.5 sources, sounds like 1.0.2 is expected. I known a way to build python with our own libssl, but where to install it? in the same prefix than cura-build-environment?

For the record, output of fist error while building Python:

/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c: In function 'newPySSLSocket':                 
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:527:9: warning: implicit declaration of function 'CRYPTO_add' [-Wimplicit-function-declaration]                                                                            
         CRYPTO_add(&inbio->bio->references, 1, CRYPTO_LOCK_BIO);                                                          
         ^~~~~~~~~~                                                                                                      
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:527:31: error: dereferencing pointer to incomplete type 'BIO {aka struct bio_st}'                                                                                           
         CRYPTO_add(&inbio->bio->references, 1, CRYPTO_LOCK_BIO);                                                             
                               ^~
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:527:48: error: 'CRYPTO_LOCK_BIO' undeclared (first use in this function)
         CRYPTO_add(&inbio->bio->references, 1, CRYPTO_LOCK_BIO);
                                                ^~~~~~~~~~~~~~~
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:527:48: note: each undeclared identifier is reported only once for each function it appears in
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c: In function '_create_tuple_for_X509_NAME':
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:739:35: error: dereferencing pointer to incomplete type 'X509_NAME_ENTRY {aka struct X509_name_entry_st}'
             if (rdn_level != entry->set) {
                                   ^~
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c: In function '_get_peer_alt_names':
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:854:16: error: dereferencing pointer to incompl
ete type 'X509_EXTENSION {aka struct X509_extension_st}'
         p = ext->value->data;
                ^~
/root/sources/cura-build-environment/build/Python-prefix/src/Python/Modules/_ssl.c:926:17: warning: 'ASN1_STRING_data' is deprecat
ed [-Wdeprecated-declarations]
                 v = PyUnicode_FromStringAndSize((char *)ASN1_STRING_data(as),
                 ^

[LipuFei]

Hi @asyd , I think you are right. Here is a bug report on this issue for python 3.5.2: https://bugs.python.org/issue29027. Looks like the problem will go away if we upgrade to 3.5.3+

[LipuFei]

@asyd I have just tried compiling python 3.5.4 with a debian:stable docker container and it did generate those SSL compile errors. You can change the python package version in projects/python.cmake to use this workaround. I have made a PR to upgrade python to 3.5.4 earlier https://github.com/Ultimaker/cura-build-environment/pull/28

[LipuFei]

Hi @asyd , and also check this PR https://github.com/Ultimaker/cura-build-environment/pull/20. I'm not sure if this can happen in a docker container or not, but you have an Intel KabyLake CPU or so, you will also need use a newer version of OpenBLAS.

[asyd]

Thanks for the feedback! I'll give a new try :)

[thopiekar]

See #34

[LipuFei]

Closing this because we use pip install for six and many other python packages now in master.