executables missing after `python setup.py install`

[tylernisonoff]

after attempting to install hsds, If I attempt to load anything using hsload I see:

/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-servicenode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-rangeget': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
/usr/local/Cellar/python@3.9/3.9.9/bin/python3.9: can't open file '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin/hsds-datanode': [Errno 2] No such file or directory
ERROR 2022-01-19 12:51:59,864 hsdsapp.py:224 failed to initialize after 10.0 seconds
ERROR 2022-01-19 12:51:59,866 hsload.py:310 Error creating file /dataset/test.hdf5: failed to initialize after 10.0 seconds

I tried uninstalling and reinstalling with various python versions, but all give me the same result.

[jreadey]

You did a python setup.py install?

After installing the hsds package, what happens when you run: ./runall.sh --no-docker?

[tylernisonoff]

You did a python setup.py install?

Correct. This is on Mac Big Sur, I can try a docker environment to rule out OS issues.

After installing the hsds package, what happens when you run: ./runall.sh --no-docker?

I have not gotten ./runall.sh --no-docker to succeed -- I get the following output indefinitely.

AWS_S3_GATEWAY set, using admin/docker/docker-compose.aws.yml
no docker startup
starting head node
starting service node
starting data node
1: waiting for server startup (status: 000)
2: waiting for server startup (status: 000)
3: waiting for server startup (status: 000)
4: waiting for server startup (status: 000)
5: waiting for server startup (status: 000)
6: waiting for server startup (status: 000)

[jreadey]

That's odd... looks like it's trying to use docker compose. Do you have the most recent code? This is the output I get (I'm on Big Sur too):

--no_docker option specified - using directory: /tmp/hs for socket and log files
using password file: admin/config/passwd.txt
AWS_S3_GATEWAY set, using mybucket S3 Bucket (verify that this bucket exists)
no docker startup
Using S3 Gateway
set logging to: 10
DEBUG:root:HsdsApp init - Using socketdir: /tmp/hs/
password_file: admin/config/passwd.txt
INFO:root:all processes ready!
INFO:root:Ready after: 0.00 s

READY! use endpoint: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock

As you can see, the output is a bit different. I'd think you'd see the print on https://github.com/HDFGroup/hsds/blob/master/runall.sh#L109: --no_docker option specified...

[tylernisonoff]

ah sorry, i tried switching to an earlier release to see if that would fix it, and forgot to switch back.

using password file: admin/config/passwd.txt
AWS_S3_GATEWAY set, using <my bucket> S3 Bucket (verify that this bucket exists)
no docker startup
Using S3 Gateway
unable to find config file
Traceback (most recent call last):
  File "/usr/local/bin/hsds", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.6.3', 'console_scripts', 'hsds')())
  File "/usr/local/lib/python3.9/site-packages/hsds-0.6.3-py3.9.egg/hsds/app.py", line 46, in main
    parser.add_argument('-p', '--port', nargs=1, default=config.get('sn_port'),
  File "/usr/local/lib/python3.9/site-packages/hsds-0.6.3-py3.9.egg/hsds/config.py", line 129, in get
    _load_cfg()
  File "/usr/local/lib/python3.9/site-packages/hsds-0.6.3-py3.9.egg/hsds/config.py", line 53, in _load_cfg
    raise FileNotFoundError(msg)

I tried passing --config admin/config/config.yml but that gave the same error

[tylernisonoff]

(I got it running on docker following the docker install instructions, but was not able to hsload )

[jreadey]

I checked in some changes that should enable config.yml to be accessed from the package install directory if it doesn't find "admin/config/config.yml". Not sure why it wasn't loading before though - you should have that file is you are launching ./runall.sh from the hsds src tree. Anyway, please try the updated code. Also, if you set an CONFIG_DIR environment variable, it will look in that directory for the config files.

What was the problem with hsload?

[tylernisonoff]

Thank you for all the help, John!

runall.sh --no-docker now starts up correctly.

running testall.py I pass most tests but one failure, however I may just need to finish the post_install setup?

OK
setup_test
testHomeFolders
req: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock/
domain: /home
/home get status: 500
got status code: 500
F
======================================================================
FAIL: testHomeFolders (__main__.SetupTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "setup_test.py", line 80, in testHomeFolders
    self.assertEqual(rsp.status_code, 200)
AssertionError: 500 != 200

----------------------------------------------------------------------
Ran 1 test in 0.014s

FAILED

Regardless, I was now able to succesfully use hsload without the missing executable output, so we can probably close this out!

[jreadey]

That's actually a bit worrisome. You shouldn't see 500 errors even without the post_install setup. Please take a look at the log file and tell me what kind of error you are getting.

[tylernisonoff]

Hi John,

Apologies for the confusion, it must have been a bad environment variable on my end. I tried them again in a clean environment and they all passed.

[jreadey]

No problem. Glad to hear everything is working for you now!