Building HSDS Service with Tencent COS

[AboPlus]

Hello,

I followed the instructions in the documentation at https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_tencent.md to set up the hsds service with Tencent COS. I have some questions that I would like to ask:

In the Sample .bashrc file, there is a configuration line: export HSDS_ENDPOINT=http://hsds.hdf.test. If I follow the instructions in the document exactly, should the value here be http://127.0.0.1:5101? After configuring HSDS_ENDPOINT as http://127.0.0.1:5101, I ran python testall.py and encountered some errors. Here is the error message:

Could you please help me identify what went wrong?

[jreadey]

Yes: http://128.0.0.1:5101 will work. I usually add a etc/hosts entry that maps hsds.hdf.test to 127.0.0.1, but that's just cosmetic.

For the test failure, could you look at the docker log files? e.g. docker logs hsds_sn_1 and note any ERROR or WARNING lines? Post those, and I'll see if I can deduce what the problem is.

[mattjala]

The docker containers need about 10-15 seconds to start up - it's possible that they were still in the process of starting up when you ran the tests. If this is the case, just running the tests again might work.

[AboPlus]

Yes: http://128.0.0.1:5101 will work. I usually add a etc/hosts entry that maps hsds.hdf.test to 127.0.0.1, but that's just cosmetic.

For the test failure, could you look at the docker log files? e.g. docker logs hsds_sn_1 and note any ERROR or WARNING lines? Post those, and I'll see if I can deduce what the problem is.

Yes, there are some WARN lines here. I have collected these WARN lines as follows:

WARN> Invalid domain: None
WARN> Invalid domain: None
WARN> Invalid domain: None
WARN> Invalid domain: None
WARN> Action: delete not permitted for user: test_user2
WARN> bucket: hsds-test-1310752370 does not exist, exception: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
WARN> bucket: hsds-test-1310752370 does not exist, exception: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
WARN> bucket: hsds-test-1310752370 does not exist, exception: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
WARN> Invalid domain: None
WARN> Invalid domain: None

I'm not sure if there's a problem with my configuration. @jreadey

[AboPlus]

The docker containers need about 10-15 seconds to start up - it's possible that they were still in the process of starting up when you ran the tests. If this is the case, just running the tests again might work.

Thank you, but I am very sure that my Docker containers have started successfully. @mattjala

[jreadey]

@AboPlus - if you do: curl http://127.0.0.1:5101/about do you get a response with "state": READY? If so, then the server has come up and the containers are talking with each other successfully.

The WARN log entries are likely due to the container not being able to see the bucket or the bucket not existing. Did you create the bucket as in step 3 of the setup instructions?

[AboPlus]

@AboPlus - if you do: curl http://127.0.0.1:5101/about do you get a response with "state": READY? If so, then the server has come up and the containers are talking with each other successfully.

The WARN log entries are likely due to the container not being able to see the bucket or the bucket not existing. Did you create the bucket as in step 3 of the setup instructions?

@jreadey Yes, when I performed the curl http://127.0.0.1:5101/about, I get a response with "state": READY.

I created the bucket successfully, And I can normally use commands such as hsls hsload hsacl hsget, but the same problem occurs with hsrm instructions.

Here is my response to executing hsrm:

(hsds3.9) [root@VM-3-133-tencentos ~/h5file]# hsrm /home/test_user1/1_static.h5
Traceback (most recent call last):
  File "/root/miniconda3/envs/hsds3.9/bin/hsrm", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/hsds3.9/lib/python3.9/site-packages/h5pyd/_apps/hsdel.py", line 158, in main
    deleteDomain(domain)
  File "/root/miniconda3/envs/hsds3.9/lib/python3.9/site-packages/h5pyd/_apps/hsdel.py", line 67, in deleteDomain
    if base_name not in hparent:
  File "/root/miniconda3/envs/hsds3.9/lib/python3.9/site-packages/h5pyd/_hl/folders.py", line 413, in __contains__
    self._getSubdomains()
  File "/root/miniconda3/envs/hsds3.9/lib/python3.9/site-packages/h5pyd/_hl/folders.py", line 317, in _getSubdomains
    raise IOError(rsp.status_code, rsp.reason)
OSError: [Errno 404] Not Found

And the docker log is as follows:

REQ> GET: / [/home/test_user1]
REQ> GET: /domains [/home/test_user1/]
WARN> bucket: resource does not exist, exception: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.

The resource here corresponds to my BUCKET_NAME configuration.

My environment variables are set as follows, I hide AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY:

export BUCKET_NAME=resource
export AWS_S3_GATEWAY=https://hsds-test-1310752370.cos.ap-beijing.myqcloud.com
export SN_PORT=5101
export HSDS_ENDPOINT=http://127.0.0.1:5101
export ADMIN_USERNAME=admin
export ADMIN_PASSWORD=admin
export LOG_LEVEL=WARN

[AboPlus]

The problem is related to the incorrect configuration of my environment variables AWS_S3_GATEWAY and BUCKET_NAME.