[Bug]: Services not starting when using Docker Compose with Git

RageBrahimi commented 3 days ago

Error Message and Logs

When using the provided Docker Compose configuration with a GitHub private repository, not all services load as expected. The same configuration works fine when executed in an empty setup (without linking to a private repository). Specifically, PostgreSQL and Redis do not seem to initialize correctly when the configuration is used with a private repository.

Docker Compose File:

services:
  directus:
    image: directus/directus:11
    volumes:
      - directus-uploads:/directus/uploads
      - directus-extensions:/directus/extensions
      - directus-templates:/directus/templates
    environment:
      - SERVICE_FQDN_DIRECTUS_8055
      - KEY=$SERVICE_BASE64_64_KEY
      - SECRET=$SERVICE_BASE64_64_SECRET
      - ADMIN_EMAIL=${ADMIN_EMAIL:-admin@example.com}
      - ADMIN_PASSWORD=$SERVICE_PASSWORD_ADMIN
      - DB_CLIENT=postgres
      - DB_HOST=postgresql
      - DB_PORT=5432
      - DB_DATABASE=${POSTGRESQL_DATABASE:-directus}
      - DB_USER=$SERVICE_USER_POSTGRESQL
      - DB_PASSWORD=$SERVICE_PASSWORD_POSTGRESQL
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - WEBSOCKETS_ENABLED=true
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8055/admin/login"]
      interval: 5s
      timeout: 20s
      retries: 10
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy

  postgresql:
    image: postgis/postgis:16-3.4-alpine
    platform: linux/amd64
    volumes:
      - directus-postgresql-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=${SERVICE_USER_POSTGRESQL}
      - POSTGRES_PASSWORD=${SERVICE_PASSWORD_POSTGRESQL}
      - POSTGRES_DB=${POSTGRESQL_DATABASE:-directus}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 20s
      retries: 10

  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - directus-redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 20s
      retries: 10

Steps to Reproduce

Use the provided Docker Compose configuration in an environment without linking to a GitHub private repository. • Result: All services (Directus, PostgreSQL, Redis) start correctly and the application works as expected.
Use the same Docker Compose configuration but link it to a GitHub private repository (e.g., for pulling custom Directus extensions or configurations). • Result: Not all services start successfully, specifically PostgreSQL and Redis fail to initialize.

Example Repository URL

No response

Coolify Version

v4.0.0-beta.360

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 22.04

Additional Information

No response

peaklabs-dev commented 3 days ago

Developer note (for confirmed bugs).

The problem is that the magic environment variables do not work with the docker compose deploy type when using git, so variables like SERVICE_PASSWORD_POSTGRESQL are empty.

[ ] Fix: Magic environment variables should also work with git deployments based on docker compose.
[ ] Fix: Magic environment variables should also work with git deployments based on docker images.

You can easily reproduce this issue by using the following public repository URL https://github.com/peaklabs-dev/coolify-test/ and the compose named magic.yaml.

Apfelwurm commented 1 day ago

@peaklabs-dev i'm looking into coolify currently and i'm very happy about its featureset and the fact that its based on laravel (most known framework for me). Since i encountered the same problem as OP, i was glad to see this issue and your message. I thought i'm going to take a look if i can find the code parts that need changes quickly. Then i found out, that the handling of the whole docker compose deploy type using git is different than the manual one, since this uses an Application rather than a Service. Since i don't have too much time to dig deeper currently, is there a reason for that difference or is is just legacy code vs new code? :)

Thank you :)

peaklabs-dev commented 1 day ago

@Apfelwurm It has some reasons, but we need to refactor that a lot and maybe it is possible to unify it into one, need to think and plan that a lot. But service templates have some differences, so it is separated for now, but unifying it would definitely be something we want to do in the future.

Apfelwurm commented 1 day ago

Thanks for the super fast answer and your awesome work :) Okay, something like that was my fear when i saw that :D Do you think its possible and with how much friction to "just" make the magic env resolving work for the application based docker containers? just so i know if it even does make sense for me to sit down and try to implement it there. Thanks :)

coollabsio / coolify