Error compiling Python 3.6 on SL6

[marcovanleeuwen]

Dear all,

I get an error message when aliBuild it trying to compile Python 3.6 on an SL6 machine. We use a Python 2.7 module to run aliBuild; I can also get Python 3.5 via modules, but not 3.6.

The error message in the log file is:

Fatal Python error: Py_Initialize: Unable to get the locale encoding File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123 raise CodecRegistryError,\ ^ SyntaxError: invalid syntax

Current thread 0x00002aaaac6b8ce0 (most recent call first): /data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.8-1/Python.sh: line 62: 5294 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9'

---

Are there any suggestions for how to address/fix this? Can I ask aliBuild not to compile Python 3.6? The package that I am trying to build at the moment is pythia.sh (PYTHIA8).

Marco.

[marcovanleeuwen]

After some googling, I found that this error can be avoided by doing 'unset PYTHONPATH' before invoking alibuild. I am not sure what this does, but in the post that I found it was suggested that that if PYTHONPATH is set (incorrectly?), python may be picking up files from a different python installation.

Is there a way to deal with this in the build recipe? E.g. by unsetting PYTHONPATH or by setting it to point to the new Python install?

[marcovanleeuwen]

Dear all,

Someone else also ran into this problem and reported it on the mailing list. I don't know enough about Python to judge whether this simple solution is the proper one. Any comments from the Python afficionados?

Marco.

[ktf]

What is not clear to me is why do you need python2.7 at all. Can't you simply run aliBuild with python 3?

[marcovanleeuwen]

Hi Guilio,

The reported problem was on a Linux system, which I think had python 2.7 as the system version. I don't know who sets the PYTHONPATH there; it may have been a module that was loaded.

On my laptop, a Macbook, I just realised that some of this was due to a virtualenv that I was using on recommendation of Dario, to avoid another issue (which I don't remember).

Python 2.7 seems to be the system version, though:
mvl@macowiec:various $ python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py
import site # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc

however, it also looks like the vanilla environment does not have $PYTHONPATH set.

Best regards,

Marco.

[ktf]

Running in a virtualenv with python2.7 and then trying to use python3 is indeed up to troubles. I myself had to redo my whole laptop setup since we moved to python3 exactly because of this issue. The only solution I've for the moment is making sure that you have aliBuild both in your python2 and python3 environments, to avoid "cross-overs".

[ktf]

Can you check if #2034 simplifies your setup / works for you?

[marcovanleeuwen]

Hi Giuilio,

I tried this, but it does quite work. However, it does work when I unset PYTHONPATH (as in the past); that may be a specific issue on this particular system, where python 2.7 is loaded as a module, and:

echo $PYTHONPATH /cm/shared/apps/python2.7/lib/python2.7

the error message that I get without unsetting pythonpath is:

/data1/software/alisoft/BUILD/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/Python

env PATH=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1/bin:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/bin:/data1/software/alisoft/slc6_x86-64/AliEn-Runtime/v2-19-le-6/bin:/data1/software/alisoft/slc6_x86-64/UUID/v2.27.1-4/bin:/data1/software/alisoft/slc6_x86-64/ApMon-CPP/v2.2.8-alice2-4/bin:/data1/software/alisoft/slc6_x86-64/libtirpc/libtirpc-1-1-4-2/bin:/data1/software/alisoft/slc6_x86-64/AliEn-CAs/v1-4/bin:/data1/software/alisoft/slc6_x86-64/OpenSSL/v1.0.2o-4/bin:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/bin:/data1/software/alisoft/slc6_x86-64/autotools/v1.5.0-1/bin:/data1/software/alisoft/slc6_x86-64/defaults-release/v1-2/bin:/nethome/leeuw179/bin:/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10/bin/:/cm/shared/apps/python2.7/bin:/cm/shared/apps/gcc/4.8.1/bin:/cm/shared/apps/sge/2011.11p1/bin/linux-x64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/sbin:/usr/sbin:/nethome/leeuw179/bin LD_LIBRARY_PATH=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1/lib:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/lib:/data1/software/alisoft/slc6_x86-64/libffi/v3.2.1-2/lib64:/data1/software/alisoft/slc6_x86-64/AliEn-Runtime/v2-19-le-6/lib:/data1/software/alisoft/slc6_x86-64/UUID/v2.27.1-4/lib:/data1/software/alisoft/slc6_x86-64/ApMon-CPP/v2.2.8-alice2-4/lib:/data1/software/alisoft/slc6_x86-64/libtirpc/libtirpc-1-1-4-2/lib:/data1/software/alisoft/slc6_x86-64/AliEn-CAs/v1-4/lib:/data1/software/alisoft/slc6_x86-64/OpenSSL/v1.0.2o-4/lib:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/lib:/data1/software/alisoft/slc6_x86-64/GCC-Toolchain/v7.3.0-alice1-4/lib64:/data1/software/alisoft/slc6_x86-64/autotools/v1.5.0-1/lib:/data1/software/alisoft/slc6_x86-64/defaults-release/v1-2/lib:/cm/shared/apps/gcc/4.8.1/lib:/cm/shared/apps/gcc/4.8.1/lib64:/cm/shared/apps/sge/2011.11p1/lib/linux-x64 PYTHONHOME=/data1/software/alisoft/INSTALLROOT/7d9fd197b2480f46ef88122c7bbc6a651322bb2f/slc6_x86-64/Python/v3.6.10-1 pip3 install certifi==2019.3.9 Fatal Python error: Py_Initialize: Unable to get the locale encoding File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123 raise CodecRegistryError, ^ SyntaxError: invalid syntax Current thread 0x00002aaaac6b9ce0 (most recent call first): /data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.10-1/Python.sh: line 66: 10514 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9'

I will now try with 'unset PYTHONPATH', which has been useful in the past.

On the Mac, I am not using virtualenv from the shell anymore; I can try/play a bit more with in the next days.

Marco.

[marcovanleeuwen]

Hi Guilio,

Compilation works now (even without 'unset PYTHONPATH')

but when using alienv enter, I get:

[leeuw179@quark ~]$ alienv enter AliRoot/latest-v5-09-53a-next-root6 Python/v3.6.10-1(33):ERROR:102: Tcl command execution failed: setenv SSL_CERT_FILE [exec python3 -c "import certifi; print(certifi.where())"]

Python/v3.6.10-1(33):ERROR:102: Tcl command execution failed: setenv SSL_CERT_FILE [exec python3 -c "import certifi; print(certifi.where())"]

Marco.

[ktf]

Can you try again with the latest fix?

[marcovanleeuwen]

Hi Giuiio,

This seems to be a step back, now I get this error while compiling:

Fatal Python error: Py_Initialize: Unable to get the locale encoding File "/cm/shared/apps/python2.7/lib/python2.7/encodings/init.py", line 123 raise CodecRegistryError,\ ^ SyntaxError: invalid syntax

Current thread 0x00002aaaac6b9ce0 (most recent call first): /data1/software/alisoft/SPECS/slc6_x86-64/Python/v3.6.10-2/Python.sh: line 66: 13537 Aborted env PATH="$INSTALLROOT/bin:$PATH" LD_LIBRARY_PATH="$INSTALLROOT/lib:$LD_LIBRARY_PATH" PYTHONHOME="$INSTALLROOT" pip3 install 'certifi==2019.3.9' ~

i.e. it seems to be loading some init.py script from the system (or rather from a previously module, which we used to get/install aliBuild itself). I think this means that the PYTHONPATH is not set correctly (maybe because I load the python module in my .bashrc ?)...

Marco.

[marcovanleeuwen]

I could compile with 'unset PYTHONPATH', but still get the same error message.

What I noticed is that there is a python3 executable in python-modules. Is that expected?

which python3 /data1/software/alisoft/slc6_x86-64/Python-modules/1.0-9/share/python-modules/bin/python3

[drsh2306]

SL6 eol 30.11.2020