[Bug]: Server token auth does not seem to work - still allows non authenticated queries

[gfox2049]

What happened?

Using token auth, rebuilt server with docker-compose with token auth settings hard coded into yml file.

Does not seem to work - still allowing client connections without token auth settings.

How to prevent allowing non-authenticated connections?

Versions

Python 3.11.4, AWS Linux (used the version from chromadb cloud formation setup.)

https://docs.trychroma.com/deployment/aws

Relevant log output

NA - works to query but it should not - seems it allows bypass of token auth even though that is hard coded into *yml

[tazarov]

hey @gfox2049, thanks for reporting this. The cloud formation template hasn't been given the proper attention for the past few months and, as such, has not been updated with the Auth.

Will it be possible for you to try the AWS terraform template, which does support auth by default and also supports the auth refresh that has been done in 0.5.0? - https://github.com/chroma-core/chroma/tree/main/examples/deployments/aws-terraform

[gfox2049]

Thanks. Tried the terraform template but could not get it to complete successfully. The Readme does not seem to reference the files in the aws-terraform directory - like chroma.tf and variables.tf. Just trying to setup token auth in chroma server. Quite urgent.

[gfox2049]

If I could get the full instructions on the aws terraform deployment, this would be very good for the adoption of chromadb for this project and all future projects with RAG + LLM. Would essentially lock in chromadb for the enterprise.

[tazarov]

@gfox2049, I have just tested the TF template, and you are right that it is dated, and some of the things are not working.

I've followed this script (as per the README.md):

git clone https://github.com/chroma-core/chroma.git chromadb
cd chromadb/examples/deployments/aws-terraform
terraform init # this should complete successfully by downloading the AWS TF provider
ssh-keygen -t RSA -b 4096 -C "Chroma AWS Key" -N "" -f ./chroma-aws && chmod 400 ./chroma-aws
export TF_VAR_AWS_ACCESS_KEY=<AWS_ACCESS_KEY>
#AWS secret access key
export TF_VAR_AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>
#path to the public key you generated above (or can be different if you want to use your own key)
export TF_ssh_public_key="./chroma-aws.pub"
#path to the private key you generated above (or can be different if you want to use your own key) - used for formatting the Chroma data volume
export TF_ssh_private_key="./chroma-aws"
#set the chroma release to deploy
export TF_VAR_chroma_release=0.5.0
# AWS region to deploy the chroma instance to
export TF_VAR_region="us-west-1"
#enable public access to the chroma instance on port 8000
export TF_VAR_public_access="true"

terraform apply -auto-approve

From the above, you'll need to replace AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY with valid credentials for AWS.

I'll fix the template asap.

[gfox2049]

Thank you. I am getting a message that it expects a Terraform init. Are there any other setup steps required. (I am running in a new clean environment).

[tazarov]

@gfox2049, fix pushed under #2302

Can you test it:

git clone -b trayan-06-06-chore_fixing_and_refreshing_aws_tf_blueprint https://github.com/chroma-core/chroma.git chroma-db-aws-fix && cd chroma-db-aws-fix/examples/deployments/aws-terraform

Let me know if you face any errors.

[gfox2049]

Looks like it worked. When I run the server does that need to be launched via docker or chroma run?

[tazarov]

@gfox2049 when you run the server, the automation will run Chroma for you in docker.

To test it out, from your TF dir.

export instance_public_ip=$(terraform output instance_public_ip | sed 's/"//g')
curl -v http://$instance_public_ip:8000/api/v1/heartbeat

[gfox2049]

Warning: No outputs found │ │ The state file either has no outputs defined, or all the defined outputs are empty.

[tazarov]

@gfox2049, did your terraform init and terraform apply complete successfully?

The apply should look like this:

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

chroma_auth_basic = <sensitive>
chroma_auth_token = <sensitive>
instance_private_ip = "172.31.18.68"
instance_public_ip = "xxx.xxx.xxx.xxx"

[gfox2049]

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

Yes - that is what it shows - with the chroma auth and instance info.

[gfox2049]

sry - see it has to be in TF dir will check

[gfox2049]

Ok. That works. So to add the auth vars I need to edit the docker *.yml and restart the instance?

[tazarov]

@gfox2049, nope you don't have to do this either. The TF template enables Auth by default. If you try:

curl -v http://$instance_public_ip:8000/api/v1/collections

You should get 403 Forbidden error.

[gfox2049]

Yes - but how do I set the values for token or user/pw?

https://docs.trychroma.com/deployment/auth#basic-authentication

[tazarov]

@gfox2049, the TF template generates a random one for you each you create a new server.

You can view it by running:

terraform output chroma_auth_token

If you want to set your own that can be added too in the template just let me know (for now it is only auto-generated).

[gfox2049]

Cannot connect to host from chroma client in python. Gets messages like server not running. Not sure the architecture here is clear in the TF example.

The earlier example below is very clear to follow the instructions. Can we get the cloud formation template updated with auth?

https://docs.trychroma.com/deployment/aws

[tazarov]

I can look into updating the aws cloud formation but it might take another day.

Can you share the stack exception and how you configure your client (code sample).

[gfox2049]

To clarify - the tf commands build the chroma server on the EC2 instance it is running in, correct? Primary question is around the public IP address. The IP from tf output does not match the IP of the EC2 instance in the console. Should it?

[gfox2049]

I've attempted executing the TF template and scripts two different ways - either from a clean, new AWS console cloud shell environment, or from a new ubuntu ec2 instance. Neither created a net new instance, but appeared to create the server from within the environment it was running. When attempting to connect to the public IP - would get the error message from the client - are you sure chromadb is running? Also - seeing in the console that no new resources are being created in ec2.

[gfox2049]

Any additional details I can provide?

[gfox2049]

Presuming that both database server and client will need to be reinstalled after recent updates?

[tazarov]

@gfox2049, if you are upgrading to 0.5.1+ then yes. Both the client and server versions need to match otherwise you'll encounter #2379