Install.sh reports it cannot find the python bindings leading to later failures

[rsdunlapiv]

Describe the bug

After running ./install_dependencies.sh && ./install.sh on my system, I see an error message that it failed to find the Python Babeltrace2 module:

/dependencies/babeltrace2-2.0.4/../INSTALL/babeltrace2-2.0.4/share/doc/babeltrace2'
make[2]: Leaving directory '/home/rocky/esmfdev/esmf-profiler.v1/dependencies/babeltrace2-2.0.4'
make[1]: Leaving directory '/home/rocky/esmfdev/esmf-profiler.v1/dependencies/babeltrace2-2.0.4'
/home/rocky/esmfdev/esmf-profiler.v1
Creating virtual environment
Failed to find Python Babeltrace2 module: /home/rocky/esmfdev/esmf-profiler.v1/dependencies/INSTALL/babeltrace2-2.0.4/lib/python3.9/site-packages
/home/rocky/esmfdev/esmf-profiler.v1/dependencies/INSTALL/babeltrace2-2.0.4/lib/python3.8/site-packages

Strangely, that path does exist:

ls /home/rocky/esmfdev/esmf-profiler.v1/dependencies/INSTALL/babeltrace2-2.0.4/lib/python3.9/site-packages
bt2  bt2-2.0.4-py3.9.egg-info

So it may be that the check itself is wrong.

This results in existing the install.sh script early, so that the LD_LIBRARY_PATH and PYTHONPATH env. variables are not appended to the venv/bin/activate script. The then leads to the test failure:

(venv) rocky@rocky-Latitude-E5570:~/esmfdev/esmf-profiler.v1$ python -m pytest
========================================================================== test session starts ===========================================================================
platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/rocky/esmfdev/esmf-profiler.v1/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/rocky/esmfdev/esmf-profiler.v1, configfile: pytest.ini, testpaths: tests
plugins: cov-3.0.0
collected 0 items / 1 error                                                                                                                                              

================================================================================= ERRORS =================================================================================
__________________________________________________________________ ERROR collecting tests/test_trace.py __________________________________________________________________
ImportError while importing test module '/home/rocky/esmfdev/esmf-profiler.v1/tests/test_trace.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_trace.py:8: in <module>
    from profiler.main import create_site_file, run_analsysis
src/profiler/main.py:16: in <module>
    from profiler.analyses import LoadBalance
src/profiler/analyses.py:1: in <module>
    from profiler.event import TraceEvent, DefineRegion, RegionProfile
src/profiler/event.py:3: in <module>
    import bt2
E   ModuleNotFoundError: No module named 'bt2'
======================================================================== short test summary info =========================================================================
ERROR tests/test_trace.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================ 1 error in 0.26s ============================================================================

To Reproduce

Run the install script. Not sure if this is an issue specific to my machine.

Expected behavior

The install.sh script should correctly locate the bt2 python bindings and not exit early.

Desktop (please complete the following information):

• OS: Ubuntu
• Version: 20.04

[rsdunlapiv]

Another important detail:

rocky@rocky-Latitude-E5570:~/esmfdev/esmf-profiler.v1$ find $PWD/dependencies/INSTALL/babeltrace2-2.0.4 -type d -name "site-packages"
/home/rocky/esmfdev/esmf-profiler.v1/dependencies/INSTALL/babeltrace2-2.0.4/lib/python3.9/site-packages
/home/rocky/esmfdev/esmf-profiler.v1/dependencies/INSTALL/babeltrace2-2.0.4/lib/python3.8/site-packages

It seems that I have bindings installed for two versions of Python3 which is probably throwing off the test. Solutions might be to install only one version or to fix the test so that it allows multiple installations. But we will have to choose only one to use for setting the environment variables.

[ryanlong1004]

@rsdunlapiv At first guess, it seems like installs are going to 3.9 before the virtual environment activation and then 3.8 after.

[rsdunlapiv]

Found it!

The INSTALL directory was incorrectly committed to the repo: https://github.com/esmf-org/esmf-profiler/tree/main/dependencies/INSTALL

This is where the extra Python install came from.

It should be removed. We will also need a v.1.1 tag since this will impact users.

[ryanlong1004]

This has been resolved in the development branch, and as @rsdunlapiv mentioned above, was caused by keeping the uncompressed dependency directories in the repo.

Removing them (leaving only the compressed files) has resolved the issue.