hatchet-dev / hatchet

A distributed, fault-tolerant task queue
https://hatchet.run
MIT License
4.05k stars 142 forks source link

Environment Variable Enhancement for `setup-config` Service in Self-Hosted Deployment #788

Open ArthurMor4is opened 4 weeks ago

ArthurMor4is commented 4 weeks ago

When deploying the application using the following docker-compose.yml:

version: "3.8"
services:
  migration:
    image: ghcr.io/hatchet-dev/hatchet/hatchet-migrate:latest
    environment:
      DATABASE_URL: "postgresql://hatchet:MY_PASSWORD@MY_RDS_HOST:5432/hatchet"
  setup-config:
    image: ghcr.io/hatchet-dev/hatchet/hatchet-admin:latest
    command: /hatchet/hatchet-admin quickstart --skip certs --generated-config-dir /hatchet/config --overwrite=false
    environment:
      DATABASE_URL: "postgresql://hatchet:MY_PASSWORD@MY_RDS_HOST:5432/hatchet?sslmode=require"
      DATABASE_POSTGRES_HOST: "MY_RDS_HOST"
      DATABASE_POSTGRES_PORT: "5432"
      DATABASE_POSTGRES_USERNAME: "hatchet"
      DATABASE_POSTGRES_PASSWORD: "MY_PASSWORD"
      SERVER_TASKQUEUE_RABBITMQ_URL: "MY_SERVER_TASKQUEUE_RABBITMQ_URL"
      SERVER_AUTH_COOKIE_DOMAIN: localhost:8080
      SERVER_AUTH_COOKIE_INSECURE: "t"
      SERVER_GRPC_BIND_ADDRESS: "0.0.0.0"
      SERVER_GRPC_INSECURE: "t"
      SERVER_GRPC_BROADCAST_ADDRESS: MY_PUBLIC_EC2_IP_ADDRESS:7077
    volumes:
      - hatchet_certs:/hatchet/certs
      - hatchet_config:/hatchet/config
    depends_on:
      migration:
        condition: service_completed_successfully

The migration service completes as expected, creating the necessary tables in my RDS instance. However, the setup-config service throws the following error:

setup-config-1      | Generating encryption keys for Hatchet server
setup-config-1      | Generating config files /hatchet/config
setup-config-1      | 2024/08/16 14:34:59 User `hatchet` was denied access on the database `hatchet.public`
setup-config-1      | panic: spawn: query engine errored: User `hatchet` was denied access on the database `hatchet.public`
setup-config-1      | 
setup-config-1      | goroutine 1 [running]:
...
service "setup-config" didn't complete successfully: exit 2

It appears that the sslmode parameter in the DATABASE_URL is being overridden or ignored. When I explicitly add the DATABASE_POSTGRES_SSL_MODE environment variable to the docker-compose.yml, the issue is resolved, and the application starts successfully. Here is the updated setup-config service configuration:

setup-config:
  image: ghcr.io/hatchet-dev/hatchet/hatchet-admin:latest
  command: /hatchet/hatchet-admin quickstart --skip certs --generated-config-dir /hatchet/config --overwrite=false
  environment:
    DATABASE_URL: "postgresql://hatchet:MY_PASSWORD@MY_RDS_HOST:5432/hatchet"
    DATABASE_POSTGRES_SSL_MODE: "require"
    DATABASE_POSTGRES_HOST: "MY_RDS_HOST"
    DATABASE_POSTGRES_PORT: "5432"
    DATABASE_POSTGRES_USERNAME: "hatchet"
    DATABASE_POSTGRES_PASSWORD: "MY_PASSWORD"
    SERVER_TASKQUEUE_RABBITMQ_URL: "MY_SERVER_TASKQUEUE_RABBITMQ_URL"
    SERVER_AUTH_COOKIE_DOMAIN: localhost:8080
    SERVER_AUTH_COOKIE_INSECURE: "t"
    SERVER_GRPC_BIND_ADDRESS: "0.0.0.0"
    SERVER_GRPC_INSECURE: "t"
    SERVER_GRPC_BROADCAST_ADDRESS: MY_PUBLIC_EC2_IP_ADDRESS:7077
  volumes:
    - hatchet_certs:/hatchet/certs
    - hatchet_config:/hatchet/config
  depends_on:
    migration:
      condition: service_completed_successfully

Since my RDS instance requires SSL mode, adding the DATABASE_POSTGRES_SSL_MODE environment variable resolved the issue.

I believe updating the documentation or the default environment variables for setup-config could help others avoid this issue.

Thank you for your consideration!

abelanger5 commented 4 weeks ago

Hey @ArthurMor4is, really appreciate the report, we'll get this into the self-hosted documentation in an upcoming release!