Server 500 due to missing keys in config.yml

[mkatgert]

We're running an instance of the HSDS server, set up according to the instructions in the readme. Almost all calls work, only the datasets/{id}/value endpoint resulted in a 500 response from the server. This happened both on a local instance running in 'Posix'-mode, as well as an instance running on AWS EC2 serving data from S3.

Steps to reproduce:

1. Setup a local running version of the server according to readme
2. Add data to the server with hsload according to readme
3. Query datasets/{id}/value endpoint.

Upon investigation of the server logs we found out that this is caused by two missing keys in the admin/config/config.yml file. These keys are aws_lambda_threshold and aws_lambda_chunkread_function. Adding these two keys with bogus values to the config file results in a fully functioning server. It might be a good idea to add these keys in the repository version, so that other people do not stumble upon this issue.

[kerberizer]

Can confirm both the problem and the solution. Just a note as well that aws_lambda_threshold apparently (and unsurprisingly too) is expected to be int, so null doesn't work, for example.

[jreadey]

Thanks for reporting this.
Those config values were removed recently since they are no longer used in the code. Did you build the container image yourself, or are you using an image from Dockerhub? Using the newer config.yml with an older image version would cause an error.

[mkatgert]

I cloned the master branch of the repository to my local computer and set up everything according to the readme. The (incomplete) config.yml can be found at https://github.com/HDFGroup/hsds/blob/master/admin/config/config.yml.

[kerberizer]

Thank you, @jreadey ! My setup is also mostly based on the readme. It does use the master branch (in terms of config.yml and the docker-compose examples) and latest from Docker Hub. If I understand correctly that this is incompatible, may I suggest, off the top of my head, one of the following as a possible way to avoid other people getting confused:

• introducing a stable branch here that would correspond to the latest Docker tag (if it's supposed to point to the latest stable version);
• repurposing the latest Docker tag to correspond to the master branch here (and pushing it regularly);
• updating the docker-compose examples in hsds/admin/docker to use a tag that is compatible with master;
• emphasizing that people should not use the pre-built images with master, and should instead build them on their own.

[jreadey]

@kerberizer, thanks for your suggestions. I think it's a convenience for users to not necessarily have to build the docker image to deploy HSDS, so be able to work with the hdfgroup/hsds:latest image in DockerHub would be preferable. (though for production deployments it's usually best to use a specific version tag).

I've added the old keys back to the config.yml file (marked as deprecated). That should enable the older container images to run with the newer config.yml. Let me know if that works for you.

[kerberizer]

I've added the old keys back to the config.yml file (marked as deprecated). That should enable the older container images to run with the newer config.yml. Let me know if that works for you.

@jreadey Thank you! Works perfectly fine for me with c7afc3dc5ab25c4f10bb7735e52c021b98975aff and, indeed, might be the best solution overall.

[jreadey]

Great! - I'll close this issue then. @mkatgert - please re-open if this update isn't working for you.

[mkatgert]

Great, this indeed fixes the issue. Thanks.