Closed TManhente closed 2 years ago
The cmd_dir
does not take into account the user local install case that is not a virtual environment. I think the fix is to first look in the os.path.join(site.getuserbase(), "bin")
folder and, if not found, in os.path.join(sys.exec_prefix, "bin")
. This should take care of all the possibilities.
I think the fix is to first look in the
os.path.join(site.getuserbase(), "bin")
folder and, if not found, inos.path.join(sys.exec_prefix, "bin")
. This should take care of all the possibilities.
I think it would work for the PIP user install case, but for the setup.py install --prefix
it would also need that the PYTHONUSERBASE
environment variable is set.
Aside from that, the idea of HSDS code needing to look for its own executables somewhere seems a little bit off. Wouldn't it be possible to the hsds
executable to pass its own location to hsds_app.py
? Something like:
# bin/hsds:
from pathlib import Path
if __name__ == '__main__':
hsds_bin_dir = Path(__file__).parent
os.environ['HSDS_BIN_DIR'] = str(hsds_bin_dir) # Don't know if there's a better way to pass this down to hsds_app
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')()
)
# lib/.../hsds/hsds_app.py:
cmd_dir = os.environ['HSDS_BIN_DIR']
pargs = [py_exe,
os.path.join(cmd_dir, "hsds-datanode"),
f"--log_prefix=dn{node_number+1} "]
Another option, considering that the HSDS executables in the bin
folder look like simple shortcuts for Python files in the lib
folder, would be hsds_app.py
running those datanode.py
, servicenode.py
etc. files directly using its own location as reference (as they are all side-by-side):
# lib/.../hsds_app.py:
from pathlib import Path
cmd_dir = Path(__file__).parent
pargs = [py_exe,
os.path.join(cmd_dir, "datanode.py"),
f"--log_prefix=dn{node_number+1} "]
The problem with the last suggested approach is that you get relative import errors. E.g.:
$ python hsds/datanode.py
Traceback (most recent call last):
File "/Users/john/projects/hsds/hsds/datanode.py", line 19, in <module>
from . import config
ImportError: attempted relative import with no known parent package
I don't see any easy way to avoid that, so I'll try out using Path(file,).parent
@TManhente - I gave @ajelenak approach a try. Can you see if the latest in this branch: https://github.com/HDFGroup/hsds/tree/nodocker works for you?
@jreadey: I've tried it with a PIP user install from your Git commit:
$ python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8
and it worked: The can't open file '/usr/bin/hsds-*
errors are gone.
However, if I try a system install, PIP seems to install the executables to /usr/local/bin
. When I tried to run those, I got the following errors:
$ sudo python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8
[...]
Successfully installed hsds-0.7.0b11 ...
$ which hsds
/usr/local/bin/hsds
$ hsds --loglevel DEBUG --root_dir /tmp/hsds-root
DEBUG:root:sys bin_dir: /usr/bin
INFO:root:no userbase or syspath found - using: /usr/local/lib/python3.6/site-packages/hsds
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-rangeget': [Errno 2] No such file or directory
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-servicenode': [Errno 2] No such file or directory
/bin/python3: can't open file '/usr/local/lib/python3.6/site-packages/hsds/hsds-datanode': [Errno 2] No such file or directory
In your first reply, you've mentioned that
The problem with the last suggested approach is that you get relative import errors.
Did you also consider the first approach I've mentioned (the hsds
executable to get its own location and pass it down to the hsds_app.py
)? Is it also not feasible, or has it any drawbacks?
What does this code reports for your system install case?
from shutil import which
for s in ('rangeget', 'servicenode', 'datanode'):
print(f"hsds-{s} --> {which(f'hsds-{s}')}")
@ajelenak: If I do a PIP user install and run Python with the standard PATH
(which doesn't contain ~/.local/bin
), the output is:
hsds-rangeget --> None
hsds-servicenode --> None
hsds-datanode --> None
If I add ~/.local/bin
to the PATH
, the output is:
hsds-rangeget --> /home/testuser/.local/bin/hsds-rangeget
hsds-servicenode --> /home/testuser/.local/bin/hsds-servicenode
hsds-datanode --> /home/testuser/.local/bin/hsds-datanode
If I do a PIP system install, the output is:
hsds-rangeget --> /usr/local/bin/hsds-rangeget
hsds-servicenode --> /usr/local/bin/hsds-servicenode
hsds-datanode --> /usr/local/bin/hsds-datanode
I've ran those tests in a CentOS 7.9 machine using is standard Python 3 from YUM (version 3.6.8) and the HSDS version from https://github.com/HDFGroup/hsds/tree/nodocker:
$ sudo yum install git python3-pip
$ sudo python3 -m pip install --upgrade pip
# User install, for system install add `sudo`:
$ python3 -m pip install git+https://github.com/HDFGroup/hsds.git@8ca732e05ff2b18500f43fc0c4671405a5cb50e8
Thanks @TManhente! The user install case is covered by os.path.join(site.getuserbase(), "bin")
. On my macOS laptop, it reports ~/.local/bin
without specifically setting any of the environment variables.
It looks like we may have a solution. The search for installed hsds command-line tools (console script entry points) will make checks in this order:
os.path.join(site.getuserbase(), "bin")
folderos.path.join(sys.exec_prefix, "bin")
foldershutil.which(<HSDSTOOL>)
Acceptable?
@TManhente Can you please try the nodocker branch again? Commit 4d8418a8e9fc2369ceba91b545e4c6b4c78f9374. I added the shutil.which
option as well.
@ajelenak, just tried it and it worked fine 👍
I followed the same steps I've mentioned in my last reply, I just had to update the Python version, as CentOS 7.9 standard python3
package is Python 3.6, which doesn't have multiprocessing.shared_memory
.
Excellent! Thanks @TManhente. Worked for me in a clean conda environment as well.
I think this is fixed now. Close the issue?
Running
hsds
from a user install (i.e. outside default system folder like/usr
) fails because it tries to run commands from the wrong path (/usr/bin
).This seems to happen both when using a user install from PIP or by specifying a
--prefix
to thesetup.py install
.PIP user install:
setup.py install --prefix:
The cause seems to be the way
cmd_dir
is computed inHsdsApp.run()
: