Local ./runall.sh - login not working for hsconfigure

[ryantoussaint]

Hi,

I am attempting to setup hsds locally, link it to an AWS S3 bucket I have created, and then use h5pyd to add content to my file and have it reflected in S3.

I have hsds running locally via $ ./runall.sh --no-docker and see the output of READY! use endpoint: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock

(Note: I ran into some trouble doing this initial setup, while on master, so I switched to https://github.com/HDFGroup/hsds/commit/4d8418a8e9fc2369ceba91b545e4c6b4c78f9374 based on https://github.com/HDFGroup/hsds/issues/122, which enabled ./runall.sh --no-docker to succeed)

I am now following https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md, and trying to run:

$ hsconfigure
$ hstouch -u admin -p admin /home/

However, hsconfigure (shown below) currently returns Unauthorized.

I have set some environment variables:

export BUCKET_NAME=XXXXX 
export AWS_S3_GATEWAY=http://s3.amazonaws.com 
export HSDS_ENDPOINT=http+unix://%2Ftmp%2Fhs%2Fsn_1.sock
export HS_USERNAME=admin
export HS_PASSWORD=admin

When I run $ hsconfigure, it fails at the end with Unauthorized (username/password or api key not valid)

$ hsconfigure
Enter new values or accept defaults in brackets with Enter.

Server endpoint [http+unix://%2Ftmp%2Fhs%2Fsn_1.sock]:
Username [admin]:
Password [admin]:
API Key [None]:
Testing connection...
Unauthorized (username/password or api key not valid)
Quit? (Y/N)

I have cp the basic admin/config/passwd.txt file, prior to running ./runall.sh --no-docker, and referenced https://github.com/HDFGroup/hsds/issues/53, but seems like a slightly different use case.

Any advice would be appreciated! In addition, do you have any links/setup guides for running HSDS locally while connecting it to s3?

I have been referencing https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_aws.md and piecing it together.

Appreciate all the documentation in general, thanks!

[jreadey]

I suspect HSDS is not finding the password file. When running in an Anaconda environment the server seems to pick up the password file using a relative path: "admin/config/passwd.txt". But I guess you are not using conda, since you had problems prior to switching to the nodocker branch.

Try this - edit the runall.sh and change the line with export PASSWORD_FILE="admin/config/passwd.txt" to use an absolute path. Let me know if that works.

Running locally while connecting to S3 should be no problem. As long as you have AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_S3_GATEWAY set as environment variables, the server should use S3 rather than posix storage.

[ryantoussaint]

Hi @jreadey thanks for the quick reply.

Unfortunately, it's the same error.

I updated PASSWORD_FILE as well as CONFIG_DIR to be absolute paths. I also updated runall.sh to print out the underlying hsds command that is being run (I'm including the whole console output for reference below):

./runall.sh --no-docker
--no_docker option specified - using directory: /tmp/hs for socket and log files
using password file: /Users/XXXX/YYYY/ZZZZ/hsds/admin/config/passwd.txt
AWS_S3_GATEWAY set, using AAAAA S3 Bucket (verify that this bucket exists)
no docker startup
Using S3 Gateway
COMMAND -----> hsds --bucket_name AAAAA --password_file /Users/XXXX/YYYY/ZZZZ/hsds/admin/config/passwd.txt --logfile hs.log  --socket_dir /tmp/hs --loglevel INFO --config_dir=/Users/XXXX/YYYY/ZZZZ/hsds/admin/config --count=4
set logging to: 20
INFO:root:using cmd_dir: /usr/local/bin
INFO:root:all processes ready!
INFO:root:Ready after: 0.00 s

READY! use endpoint: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock

In the separate terminal, the hsconfigure is the same:

$ hsconfigure
Enter new values or accept defaults in brackets with Enter.

Server endpoint [http+unix://%2Ftmp%2Fhs%2Fsn_1.sock]:
Username [admin]:
Password [admin]:
API Key [None]:
Testing connection...
Unauthorized (username/password or api key not valid)
Quit? (Y/N)

Thanks

[jreadey]

Ok - it was worth a try!

If you look at the server log file (/tmp/hs/hs.log), do you see a line like: sn INFO> Loading password file: /Users/XXXX/YYYY/ZZZZ/hsds/admin/config/passwd.txt ?

[ryantoussaint]

Yes I do

...
...
sn INFO> CORS add route: <ResourceRoute [PUT] <DynamicResource  /datasets/{id}/acls/{username}> -> <function PUT_ACL at 0x117220af0>
sn INFO> CORS add route: <ResourceRoute [GET] <DynamicResource  /datasets/{id}/acls> -> <function GET_ACLs at 0x117220a60>
sn INFO> Using metadata memory cache size of: 134217728
sn INFO> allow_noauth = True
sn INFO> initUserDB
sn INFO> Loading password file: /Users/XXXX/YYYY/ZZZZ/hsds/admin/config/passwd.txt
sn INFO> using password file: /Users/XXXX/YYYY/ZZZZ/hsds/admin/config/passwd.txt
sn INFO> added user: admin
sn INFO> added user: test_user1
sn INFO> added user: test_user2
sn INFO> user_db initialized: 3 users
sn INFO> initgroupDB
sn INFO> No groups file
sn INFO> group_db initialized: 0 groups
binding to socket: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock
======== Running on http://unix:/tmp/hs/sn_1.sock: ========
(Press CTRL+C to quit)
dn1 INFO> s3sync nothing to update
dn2 INFO> s3sync nothing to update
dn3 INFO> s3sync nothing to update
dn4 INFO> s3sync nothing to update
dn1 INFO> s3syncCheck no objects to write, sleeping for 1
dn2 INFO> s3syncCheck no objects to write, sleeping for 1
dn3 INFO> s3syncCheck no objects to write, sleeping for 1
dn4 INFO> s3syncCheck no objects to write, sleeping for 1
dn1 INFO> s3sync nothing to update
dn2 INFO> s3sync nothing to update
...

Here is the contents of passwd.txt

# HSDS password file template
#
#
# This file contains a list of usernames/passwords that will be used to authenticate
# requests to HSDS.
# If using HTTP Basic Auth, copy file to "passwd.txt" in the same directory before deploying HSDS.
# Otherwise, if using Azure Active Directory or Kerberos, don't copy this file - usernames will be
# authenticated using those identity providers.
# For production use, replace the "test" password below with secret passwords and add
# and any new accounts desired.
admin:admin
test_user1:test
test_user2:test

[jreadey]

What do you get when you run the following:

$ hsinfo -e http+unix://%2Ftmp%2Fhs%2Fsn_1.sock -u test_user1 -p test

?

[ryantoussaint]

$ hsinfo -e http+unix://%2Ftmp%2Fhs%2Fsn_1.sock -u test_user1 -p test
server name: Highly Scalable Data Service (HSDS)
server state: READY
endpoint: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock
username: test_user1
password: ****
home: NO ACCESS
server version: 0.7.0beta
node count: 4
up: 11 min 11 sec
h5pyd version: 0.9.2

[jreadey]

That's an improvement over unauthorized! So looks like the password is getting loaded correctly. Do you get the same result without the mod to runall.sh?

The NO ACCESS comes from the server looking for a folder "/home/test_user1/" and not finding one. You can set the folder up as described in steps 3 and 4 of https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md.

[ryantoussaint]

Cool, same output for hsinfo after setting the paths back to relative in runall.sh! 👍

You can set the folder up as described in steps 3 and 4 of https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md

Yeah the post_install was what led me to the issue with hsconfigure -- planning to complete those steps after resolving this.

Appreciate the help!

[jreadey]

awesome - glad we got it sorted out!

[ryantoussaint]

@jreadey apologies for the confusion -- I still have the issue with hsconfigure

$ hsconfigure
Enter new values or accept defaults in brackets with Enter.

Server endpoint [http+unix://%2Ftmp%2Fhs%2Fsn_1.sock]:
Username [admin]:
Password [admin]:
API Key [None]:
Testing connection...
Unauthorized (username/password or api key not valid)
Quit? (Y/N)

Given that I have two terminals -- one running ./runall.sh --no-docker and the other testing hsconfigure, do I need to run anything additional in the second terminal to pick up any of the changes that we're making ./runall.sh --no-docker (I don't believe I have to, but thought I would confirm)?

[jreadey]

No, the other terminal just needs to know the endpoint, username, and password.

[ryantoussaint]

@jreadey do you have any other recommendations on what to try, to resolve the Unauthorized (username/password or api key not valid) from hsconfigure?

It's interesting that hsinfo -e http+unix://%2Ftmp%2Fhs%2Fsn_1.sock -u test_user1 -p test looks valid, but hsconfigure still fails.

[jreadey]

Looks like hsconfigure was treating the input text "None" as a API_key rather than no api key. I have a fix checked into the master branch of h5pyd if you want to try it out.

[jreadey]

@ryantoussaint - I'm going to close this issue. Please re-open if you are still having issues with hsconfigure.

[jreadey]

Changes are merged into master branch