Better Python Handling

[bjoo]

Currently there seems to be an issue if Python libraries are in a nonstandard place. An example is NERSC, where the python3 environment is an ANACONDA environment created by 'conda'. In this instance libraries are in ${CONDA_PREFIX}/lib, includes are in ${CONDA_PREFIX}/include and python3 itself is in ${CONDA_PREFIX}/bin.

These can be specified to CMake via -DPYTHON_EXECUTABLE= ... -DPYTHON_INCLUDE_DIR=... and -DPYTHON_LIBRARY=... with appropriate PATHs.

However, in QPhiX the codegen is built by an entirely independent sub-build which is not currently passed these variables from the calling build. The codegen build then doesn't find Pyton and fails:

INFO,Looking for Python CMake Error at /global/common/cori/software/cmake/3.5.2/carl/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message): Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "3.0") Call Stack (most recent call first): /global/common/cori/software/cmake/3.5.2/carl/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE) /global/common/cori/software/cmake/3.5.2/carl/share/cmake-3.5/Modules/FindPythonLibs.cmake:265 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) CMakeLists.txt:123 (find_package)

A quick fix is to build codegen separately, and pass it the above PYTHON variables. Then build QPhiX using -DQPHIX_CODEGEN=... to specify the location for the Codegen Lib. For a scripted build like Chroma this is not an issue.

An alternative fix, which may allow an all in one build even in these case, is for QPhiX CMake to check for Python, establish these values and explicitly pass them as -D options to the sub-build. If Python is not found one will know earlier on, rather than just at the codegen build stage. This would probably be a better solution.

[martin-ueding]

I'd think if one correctly sets the PATH and the PYTHONPATH and (perhaps LD_LIBRARY_PATH) to proper values, it should work fine. It looks like a symptom of an incorrect Python installation if calling python3 in the interactive shell does not start the Python interpreter with access to the installed libraries.

I'd say this is a “won't fix” for this project but rather something for this particular environment.

[martin-ueding]

Why is the shebang of codegen/jinja/generate_files.py changed to #!/usr/bin/env @PYTHON_EXECUTABLE@? That looks like some Autotools replacement. I guess that can be reverted back to python3?

[bjoo]

The @PYTHON_EXECUTABLE@ should be replaced by cmake for whatever python executable the FindPython test finds. It seems some extra effort is needed on some systems e.g. our interactive nodes ( I installed python 3 in my homedir) and NERSC (use anaconda & python3). On NERSC, the error messages complained about not finding the libs. It maybe that the subshell for the sub cmake didnt inherit the conda environment? These are both crucial systems for me. So I will figure out a fix for this. However, the fixes shouldnt affect systems with python in more regular places. The -D options I mentioned earlier are the ones that are user definable for the CMake FindPython call.

On June 17, 2017 4:38:46 AM EDT, Martin Ueding notifications@github.com wrote:

Why is the shebang of codegen/jinja/generate_files.py changed to #!/usr/bin/env @PYTHON_EXECUTABLE@? That looks like some Autotools replacement. I guess that can be reverted back to python3?

-- You are receiving this because you were assigned. Reply to this email directly or view it on GitHub: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_qphix_issues_62-23issuecomment-2D309202203&d=DwICaQ&c=lz9TcOasaINaaC3U7FbMev2lsutwpI4--09aP8Lu18s&r=SC-qvz5njMoFH6cliT5XZQ&m=lrBQ4axYTxs-yk10Mp_UtMgOaAkZftoXEsMrO-bo17k&s=gC-thTTJenNTnIDYcfwBKNp7MxrQHIqB9AHQ_-Chlg4&e=

-- Bálint Joó, Scientific Computing Group, Jefferson Lab Email: bjoo@jlab.org Tel: +1 757 269 5339 Sent from a mobile device

[bjoo]

Should be fixed by b7a6895abe6c144cf29c5731495391193f891342 (let's see if Travis agrees with me)

[bjoo]

Travis build succeeded so I guess this is good. I'll close this now.