Wrong executable paths when run in a user install

[TManhente]

Running hsds from a user install (i.e. outside default system folder like /usr) fails because it tries to run commands from the wrong path (/usr/bin).

/usr/bin/python3: can't open file '/usr/bin/hsds-servicenode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-rangeget': [Errno 2] No such file or directory

This seems to happen both when using a user install from PIP or by specifying a --prefix to the setup.py install.

PIP user install:

$ python3 -m pip install --user hsds
[...]
Successfully installed hsds-0.7.0b11 ...
$ ls ~/.local/bin
hsds  hsds-datanode  hsds-headnode  hsds-rangeget  hsds-servicenode  ...
$ export PATH="${PATH}:${HOME}/.local/bin"
$ hsds --root_dir /tmp/hsds-root
/usr/bin/python3: can't open file '/usr/bin/hsds-servicenode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-rangeget': [Errno 2] No such file or directory

setup.py install --prefix:

$ python3 setup.py install --prefix /tmp/hsds-install
$ ls /tmp/hsds-install/bin/
hsds  hsds-datanode  hsds-headnode  hsds-rangeget  hsds-servicenode  ...
$ export PATH="${PATH}:/tmp/hsds-install/bin"
$ export PYTHONPATH="/tmp/hsds-install/lib/python3.8/site-packages"
$ hsds --root_dir /tmp/hsds-root
/usr/bin/python3: can't open file '/usr/bin/hsds-servicenode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/bin/python3: can't open file '/usr/bin/hsds-rangeget': [Errno 2] No such file or directory

The cause seems to be the way cmd_dir is computed in HsdsApp.run():

cmd_dir = os.path.join(sys.exec_prefix, "bin")

[ajelenak]

The cmd_dir does not take into account the user local install case that is not a virtual environment. I think the fix is to first look in the os.path.join(site.getuserbase(), "bin") folder and, if not found, in os.path.join(sys.exec_prefix, "bin"). This should take care of all the possibilities.

[TManhente]

I think the fix is to first look in the os.path.join(site.getuserbase(), "bin") folder and, if not found, in os.path.join(sys.exec_prefix, "bin"). This should take care of all the possibilities.

I think it would work for the PIP user install case, but for the setup.py install --prefix it would also need that the PYTHONUSERBASE environment variable is set.

Aside from that, the idea of HSDS code needing to look for its own executables somewhere seems a little bit off. Wouldn't it be possible to the hsds executable to pass its own location to hsds_app.py? Something like:

# bin/hsds:
from pathlib import Path

if __name__ == '__main__':
    hsds_bin_dir = Path(__file__).parent
    os.environ['HSDS_BIN_DIR'] = str(hsds_bin_dir)   # Don't know if there's a better way to pass this down to hsds_app

    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')()
    )

# lib/.../hsds/hsds_app.py:
cmd_dir = os.environ['HSDS_BIN_DIR']

pargs = [py_exe,
         os.path.join(cmd_dir, "hsds-datanode"),
         f"--log_prefix=dn{node_number+1} "]

Another option, considering that the HSDS executables in the bin folder look like simple shortcuts for Python files in the lib folder, would be hsds_app.py running those datanode.py, servicenode.py etc. files directly using its own location as reference (as they are all side-by-side):

# lib/.../hsds_app.py:
from pathlib import Path

cmd_dir = Path(__file__).parent

pargs = [py_exe,
         os.path.join(cmd_dir, "datanode.py"),
         f"--log_prefix=dn{node_number+1} "]

[jreadey]

The problem with the last suggested approach is that you get relative import errors. E.g.:

$ python hsds/datanode.py
Traceback (most recent call last):
  File "/Users/john/projects/hsds/hsds/datanode.py", line 19, in <module>
    from . import config
ImportError: attempted relative import with no known parent package

I don't see any easy way to avoid that, so I'll try out using Path(file,).parent

[jreadey]

@TManhente - I gave @ajelenak approach a try. Can you see if the latest in this branch: https://github.com/HDFGroup/hsds/tree/nodocker works for you?

[TManhente]

@jreadey: I've tried it with a PIP user install from your Git commit:

$ python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8

and it worked: The can't open file '/usr/bin/hsds-* errors are gone.

However, if I try a system install, PIP seems to install the executables to /usr/local/bin. When I tried to run those, I got the following errors:

$ sudo python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8
[...]
Successfully installed hsds-0.7.0b11 ...
$ which hsds
/usr/local/bin/hsds
$ hsds --loglevel DEBUG --root_dir /tmp/hsds-root
DEBUG:root:sys bin_dir: /usr/bin
INFO:root:no userbase or syspath found - using: /usr/local/lib/python3.6/site-packages/hsds
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-rangeget': [Errno 2] No such file or directory
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-servicenode': [Errno 2] No such file or directory
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-datanode': [Errno 2] No such file or directory

In your first reply, you've mentioned that

The problem with the last suggested approach is that you get relative import errors.

Did you also consider the first approach I've mentioned (the hsds executable to get its own location and pass it down to the hsds_app.py)? Is it also not feasible, or has it any drawbacks?

[ajelenak]

What does this code reports for your system install case?

from shutil import which

for s in ('rangeget', 'servicenode', 'datanode'):
    print(f"hsds-{s} --> {which(f'hsds-{s}')}")

[TManhente]

@ajelenak: If I do a PIP user install and run Python with the standard PATH (which doesn't contain ~/.local/bin), the output is:

hsds-rangeget --> None
hsds-servicenode --> None
hsds-datanode --> None

If I add ~/.local/bin to the PATH, the output is:

hsds-rangeget --> /home/testuser/.local/bin/hsds-rangeget
hsds-servicenode --> /home/testuser/.local/bin/hsds-servicenode
hsds-datanode --> /home/testuser/.local/bin/hsds-datanode

If I do a PIP system install, the output is:

hsds-rangeget --> /usr/local/bin/hsds-rangeget
hsds-servicenode --> /usr/local/bin/hsds-servicenode
hsds-datanode --> /usr/local/bin/hsds-datanode

I've ran those tests in a CentOS 7.9 machine using is standard Python 3 from YUM (version 3.6.8) and the HSDS version from https://github.com/HDFGroup/hsds/tree/nodocker:

$ sudo yum install git python3-pip
$ sudo python3 -m pip install --upgrade pip
# User install, for system install add `sudo`:
$ python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8

[ajelenak]

Thanks @TManhente! The user install case is covered by os.path.join(site.getuserbase(), "bin"). On my macOS laptop, it reports ~/.local/bin without specifically setting any of the environment variables.

It looks like we may have a solution. The search for installed hsds command-line tools (console script entry points) will make checks in this order:

1. os.path.join(site.getuserbase(), "bin") folder
2. os.path.join(sys.exec_prefix, "bin") folder
3. shutil.which(<HSDSTOOL>)
4. Throw some form of a not found exception

Acceptable?

[ajelenak]

@TManhente Can you please try the nodocker branch again? Commit 4d8418a8e9fc2369ceba91b545e4c6b4c78f9374. I added the shutil.which option as well.

[TManhente]

@ajelenak, just tried it and it worked fine 👍

I followed the same steps I've mentioned in my last reply, I just had to update the Python version, as CentOS 7.9 standard python3 package is Python 3.6, which doesn't have multiprocessing.shared_memory.

[ajelenak]

Excellent! Thanks @TManhente. Worked for me in a clean conda environment as well.

I think this is fixed now. Close the issue?