Using google cloud bucket to run HSDS

[bilalshaikh42]

Hello,

We are trying to switch our backend from an S3 compatible Scality server to a Google Cloud bucket. However, we are getting 500 errors when attempting to post or get domains. We have set the AWS endpoint, bucket name, secret key and access key as specified by google to connect to the bucket in compatibility mode with AWS.

Do you know if there is some feature of AWS being used that would cause issues with the google bucket backend? From the logs I am getting s3 client errors when trying to read the domain.json files and an error stating that the "root dir" is not set, which seems to be what happens if the s3 connection fails based on our discussion in #107

Any insight would be much appreciated! I can also try to run the docker image locally and see if i can get some better logs to see if there is any more specific info about the issue with the connection to the bucket.

[jreadey]

The root_dir message indicates that HSDS is trying to use the posix driver, so it's not surprising that calls are failing!

The logic for switching between different storage drivers is here: https://github.com/HDFGroup/hsds/blob/master/hsds/util/storUtil.py#L92. It is looking for an S3_GATEWAY config to indicate the use of S3. Are you setting that to an endpoint valid for a Google S3 emulation?

[bilalshaikh42]

Yup, here is the only change made. From

aws_s3_gateway: http://s3low.scality.uchc.edu 

to

aws_s3_gateway: https://storage.googleapis.com 

The gateway was changed to https://storage.googleapis.com which is the s3 compatible gateway for google cloud, along with the appropriate credential change

This causes the error message about the no root_dir to pop up. I don't see any messages about a failure to connect to the bucket or s3

[bilalshaikh42]

It seems that even when I know that have the wrong credentials, I don't see any errors pertaining to s3/connecting to the bucket, so it is hard to tell what the cause may be

[bilalshaikh42]

This was again my mistake. Just like in #107 the values were not being loaded correctly. I did not realize that the data node and service node each have their own separate configurations loaded.

[jreadey]

@bilalshaikh42 - glad to hear you figured this out! Data node & service node do both pull values from the same config.yml. But these can be overriden by environment variables (which can be set per container in your deployment yaml for example).

Sometimes it's useful to just ssh into the container (kubectl exec) and checkout what the actual configuration is. Also if you run into issues, review the log files. Each container will log any configuration overrides on startup.