C++ error messages do not appear in Jupyter notebooks

[jeanfeydy]

Hi @joanglaunes, @bcharlier,

I am currently making final checks for the KeOps v2.0 release, and am noticing again a pretty cumbersome issue: C++ configuration issues are impossible to diagnose from a Jupyter notebook.

For instance, these two error messages:

[pyKeOps] Compiling nvrtc binder for python ... /vol/bitbucket/jfeydy/miniconda3/bin/python3: No module named pybind11
/data/jean/keops/pykeops/common/keops_io/pykeops_nvrtc.cpp:5:10: fatal error: pybind11/pybind11.h: No such file or directory
 #include <pybind11/pybind11.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

and

<stdin>:1:10: fatal error: cuda.h: No such file or directory
compilation terminated.
[KeOps] Warning : 
    The location of Cuda header files cuda.h and nvrtc.h could not be detected on your system.
    You must determine their location and then define the environment variable CUDA_PATH,
    either before launching Python or using os.environ before importing keops. For example
    if these files are in /vol/cuda/10.2.89-cudnn7.6.4.38/include you can do :
      import os
      os.environ['CUDA_PATH'] = '/vol/cuda/10.2.89-cudnn7.6.4.38'

[pyKeOps] Compiling nvrtc binder for python ... In file included from /data/jean/keops/pykeops/common/keops_io/pykeops_nvrtc.cpp:4:0:
/data/jean/keops/keopscore/binders/nvrtc/keops_nvrtc.cpp:6:10: fatal error: nvrtc.h: No such file or directory
 #include <nvrtc.h>
          ^~~~~~~~~
compilation terminated.
OK

are completely hidden from Jupyter notebook users (i.e. most data scientists), who only see:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-a1d3acc1b22b> in <module>
      5 print(pykeops.__version__)
      6 print(pykeops.__file__)
----> 7 pykeops.test_numpy_bindings()

/data/jean/keops/pykeops/test/install.py in test_numpy_bindings()
     19 
     20     my_conv = pknp.Genred(formula, var)
---> 21     if np.allclose(my_conv(x, y).flatten(), expected_res):
     22         pyKeOps_Message("pyKeOps with numpy bindings is working!", use_tag=False)
     23     else:

/data/jean/keops/pykeops/numpy/generic/generic_red.py in __call__(self, backend, device_id, ranges, out, *args)
    312             dtype,
    313             "numpy",
--> 314             self.optional_flags,
    315         ).import_module()
    316 

/data/jean/keops/keopscore/utils/Cache.py in __call__(self, *args)
     62                     self.library[str_id] = self.cls(params, fast_init=True)
     63                 else:
---> 64                     obj = self.cls(*args)
     65                     self.library_params[str_id] = obj.params
     66                     self.library[str_id] = obj

/data/jean/keops/pykeops/common/keops_io/LoadKeOps_nvrtc.py in __init__(self, fast_init, *args)
     12 class LoadKeOps_nvrtc_class(LoadKeOps):
     13     def __init__(self, *args, fast_init=False):
---> 14         super().__init__(*args, fast_init=fast_init)
     15 
     16     def init_phase2(self):

/data/jean/keops/pykeops/common/keops_io/LoadKeOps.py in __init__(self, fast_init, *args)
     29             self.tools = numpytools
     30 
---> 31         self.init_phase2()
     32 
     33     def init(

/data/jean/keops/pykeops/common/keops_io/LoadKeOps_nvrtc.py in init_phase2(self)
     17         import importlib
     18 
---> 19         pykeops_nvrtc = importlib.import_module("pykeops_nvrtc")
     20 
     21         if self.params.c_dtype == "float":

/vol/bitbucket/jfeydy/miniconda3/lib/python3.7/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 

/vol/bitbucket/jfeydy/miniconda3/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)
/vol/bitbucket/jfeydy/miniconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)
/vol/bitbucket/jfeydy/miniconda3/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)
ModuleNotFoundError: No module named 'pykeops_nvrtc'

in their notebooks cells when running e.g. import pykeops; pykeops.test_numpy_bindings().

In order to ease installation and configuration issues, do you think that we could redirect these "stderr" messages towards the Jupyter cells?

See you soon, Jean

[joanglaunes]

Hello Jean, I just did modifications in branch redirect_stderr to get the stderr output, using subprocess.run with the option redirect_output=True. The only problem is that this option appeared in Python 3.7 only. Do you think we can impose Python version >= 3.7 for KeOps ? Or someone know how to do without this option ?

[jeanfeydy]

Hi @joanglaunes ,

Since this is a fairly minor issue, do you think that we could simply detect the Python version and use the option if and only if Python >= 3.7? The main purpose of the fix would be to help as many users as possible to fix their configuration without creating an issue, so having a hard dependency on Python >= 3.7 would be counter-productive I think.

[joanglaunes]

Ok it's done now in master.

[jeanfeydy]

Hi @joanglaunes, Thanks a lot, I will try this today!