[Bug]: Docker-compose deployments from repositories don't work if they need additional files

[shantanugoel]

Description

Same as in issue #1805 If you make a source from a repository and use Docker Compose build pack, it does not copy any files from the repository to the application folder besides docker-compose file. Using Beta 258 Looks like there were some fixes done in Beta 233, but that only started creating the directories referred to in the docker-compose file. It does not actually copy the files in the repository to the applications folder.

Minimal Reproduction (if possible, example repository)

Create new resource Choose Public Repository Repository URL: https://github.com/authelia/authelia/ Use the following in general config: a. Build Pack: Docker compose b. Raw Compose Deployment: True (Though I've tried that this behavior remains same even for raw compose deployment being false) c. Base Directory: /examples/compose/local/ d. Docker Compose Location: /docker-compose.yml Go to the install location on disk: /data/coolify/applications/{YOUR_FOLDER} See that only README and docker-compose.yml exists, not any other configuration needed. After beta 233, it creates the additional folders (e.g. traefik and authelia) in the application folder, but does not copy the files inside them from the source repository.

Exception or Error

No response

Version

v4.0.0-beta258

[nikolouzos]

I don't see a docker-compose.yml in the repository linked (unless you are creating it in Coolify?). Have you tried using their Docker image or building the Dockerfile instead?

The volumes inside docker-compose are persistent data that are created from the image itself, it wouldn't be there unless the image has run and created data. If you need to copy files from the repository to the created image, that should happen in the Dockerfile.

[piotrprzybytniewski]

+1, tried to use a metadata/migrations directory from a Hasura repository and it only creates an empty folder, it never transfers the files from the directory itself :((

[shantanugoel]

@nikolouzos the docker-compose.yml is at https://github.com/authelia/authelia/tree/master/examples/compose/local as referenced in the base directory configuration in the original bug report. The docker-compose file is copied appropriately to the destination. I've also tried this with the docker-compose.yml being in the root of the repository as well and thus the base directory being / with the same result.

I think the issue is that coolify does not really copy any files from the source repository, but it likely parses the docker-compose files and creates the referenced directories.

[piotrprzybytniewski]

A temporary workaround for docker-compose with Coolify bug, where it doesn't copy the volume directory files, only creates an empty folder as for the version v4.0.0-beta.265 and you need some directories moved into the container:

1. ssh into your VPS
2. git clone your private repository into some folder, e.g. /home/hasura in my case
3. map this root path in your docker-compose.yml in the volumes section of the service to mount the host (server) directory into the container
4. success, after rebuild your container has those directories and files available and app in my case Hasura does run correctly with files like migrations, metadata from the repository :)

services:
   ...
    volumes:
      - /home/hasura/metadata:/hasura-metadata
      - /home/hasura/migrations:/hasura-migrations

[mcosti]

+1 on this issue

My docker-compose files are mounting the contents of the git repository on the root of the Docker image itself (to allow for no docker rebuilds if not needed)

[peyloride]

+1 on this one and this simply makes coolify unusable for me. Because, I have an rails app with docker-compose.yml file on root folder. Deploying to another VPS or local works; but when deploying to coolify using docker-compose; it simply doesn't copy the files from the repo.

[michael-at-boost]

+1 for please fix this. Also +1 for you can work around it with a Dockerfile. I'm running a build step in a composer container which starts by doing a COPY . /app to bring in all files from git. Then the CMD for that service runs first and copies the files into a dir which is mounted on all containers. Now the files exist as if we'd brought them in directly from the git repo and mounted. It's not ideal but it works.

[simonbermudez]

+1

[jamielife]

Also having this issue

[mmirca]

So glad to have found this issue, I was going over the docs and configs again and again with no success.

I think it would be really helpful to show a disclaimer in the docs. Letting the users know that the files are not automatically copied and they need to implement their own strategy for now.

Also thinking about an intuitive solution for this scenario. How about automatically copying the files for the first deployment and adding a checkbox in the persistent storage tab that allows the user to mark what should be overwrittable?

By default it would not be checked to avoid losing important data.

[zanzlender]

Another use-case from my side.

I want to add a new Resource to my project - Glances for monitoring - using their docker-compose. I go to My project > New Resource > Docker compose.

The problem is that the docker-compose file requires a glances.conf config file to work at all, and ideally a Dockerfile (although it can do without it). However, there is no way to include the config file somewhere.

I though of 2 possible solutions but they both seem to be a lot of hassle for me to consider them a long-term solution:

1. Create my own separate repo just for to have the docker-compose and config file (not tested, but idk if it will work because of OP)
2. Have a MinIO storage service and store it there - but can you access it then?

[johnpccd]

+1

[martmalo]

+1

[replete]

I spent all day trying to figure out why my files were missing, so a +1 for me with v4.0.0-beta.306.

I set up a docker-compose git app project, and the issue is that the app folder /data/coolify/applications/abcdef/ never gets the files from git. The only files that make it there come from the image and the parsed docker-compose files.

I fully expected my app source files to be copied over from git, I'm assuming this is a bug but it might be a feature?

If you're docker compose includes a volume to mount /var/www/html in the container to myapp outside it, and in the repo there's a folder named myapp, I assumed it would be included so the app would work like it would locally.

If so please consider an option to enable inclusion of files/folders for the build, to make it possible to build docker-compose based apps from git which use the source files in the image build. I just realized this is the reason why I couldn't get 4 or 5 projects to work when testing out software while evaluating coolify

[michael-at-boost]

Hope this gets added/fixed soon. Until then, to expand on the fix, here's demo repo with a "copy" service that moves files from the git repo, into an image, then mounts them on all services, as if Coolify had brought everything in from git.

https://github.com/boostmetrica/coolify-count-and-mount-demo

[replete]

If there are environment variables that contain the path to the artifact folder (containing the git files? Is that right.. the folder disappears and I can't check), we might be able to use Custom Build Command to copy this over, for a simpler workaround:

Maybe something like cp -r /artifacts/"${ARTIFACT_ID}/foldername" . && docker compose build ?

I'm not sure where the git contents exist at this point. Any ideas?

[michael-at-boost]

Nice idea. Looks like it would be /artifacts/"${ARTIFACT_ID}/${GIT_REPO_NAME}" Not suggesting those vars exist, but that would be the format. https://github.com/coollabsio/coolify/blob/e9158b7305bb9253fe75a0e76ef34af62f75199b/app/Models/Application.php#L685:L695

[Lucasnribeiro]

Hope this gets added/fixed soon. Until then, to expand on the fix, here's demo repo with a "copy" service that moves files from the git repo, into an image, then mounts them on all services, as if Coolify had brought everything in from git.

https://github.com/boostmetrica/coolify-count-and-mount-demo

yep, that's the quickest solution.

[ValerioMeschi]

+1 this is a must have - isn't there something to be done with pre-deploy commands ?

[OlegWock]

This is very frustrating issue. For me this is huge block and a critical feature to have.

I'd very appreciate at least some message, so me and other users won't spend hours trying to understand what's wrong. Especially taking into consideration that volume is still present in docker-compose file after Coolify's processing, i.e. I'd expect it to work if it's still there

[replete]

If there are environment variables that contain the path to the artifact folder (containing the git files? Is that right.. the folder disappears and I can't check), we might be able to use Custom Build Command to copy this over, for a simpler workaround:

Maybe something like cp -r /artifacts/"${ARTIFACT_ID}/foldername" . && docker compose build ?

I'm not sure where the git contents exist at this point. Any ideas?

I've been trying to find a command we can use here to achieve this, but can't find the location of the temporary git artifact folder on the filesystem. I've run inotifywait -m -r -e access,modify,create,delete ./ --format '%w%f %e' >> /home/user/inotify_output.txt on both host and coolify container to watch the filesystem and /artifacts/ doesn't seem to exist. Does anyone know where /artifacts/$UUID is created? If we can access it, we can probably do what we want in a custom build command. The debug logs are just not detailed enough for this kind of debugging.

[lnnrtngl]

Does anyone know where /artifacts/$UUID is created? If we can access it, we can probably do what we want in a custom build command. The debug logs are just not detailed enough for this kind of debugging.

I believe the /artificats/uuid directory is created within the temporarily Coolify helper container to build the containers for the service.

Also, I believe the docker rm -f container-id line is the temp helper containers' id

[replete]

Thanks @lnnrtngl, I can't test this right now, but maybe something like this would work:

docker exec $(docker ps --filter ancestor=ghcr.io/coolabsio/coolify-helper:latest --format "{{.ID}}" | head -n 1) cp -r ./app /artifacts/${ARTIFACT_ID}/${GIT_REPO_NAME} && docker compose build` 

Or maybe not, I'm guessing the helper isn't available then... will try later

[lorenzomigliorero]

+1 This is blocking my Verdaccio deployment, as it needs the config file https://github.com/verdaccio/verdaccio/tree/master/docker-examples/v4/docker-local-storage-volume

[carbontwelve]

This is still broken as of v4.0.0-beta.315. The entire git project content should be available at time of running docker compose.

[andrasbacsai]

Checking this!

[andrasbacsai]

I will add a new flag to the build phase. This will copy the Base Directory content from git to the deployment directory before deploying your application.

Hopefully this will help in your case.

The dir will look like this:

Note: Authelia uses Traefik in their compose file, which may not work with the integrated proxy of Coolify, as they both want to use ports 80 and 443.

[replete]

Going to test this as soon as the update lands. Thanks for fixing this

[goulartf]

@andrasbacsai When I selected the new option to keep the files, I got the following error:

`[2024-Jul-19 22:18:51.092462] Starting deployment of splendid-sardine-sccww40 to localhost. [2024-Jul-19 22:18:51.337747] Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:latest. [2024-Jul-19 22:18:56.207443] ---------------------------------------- [2024-Jul-19 22:18:56.212857] Importing (commit sha HEAD) to /artifacts/lggswcw. [2024-Jul-19 22:19:05.633228] Pulling & building required images. [2024-Jul-19 22:19:06.004280] Removing old containers. [2024-Jul-19 22:19:06.353648] Starting new application. [2024-Jul-19 22:19:08.006655] rm:

[2024-Jul-19 22:19:08.015507] refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/.' rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/..'

[2024-Jul-19 22:19:08.419927] rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/.' rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/..' `

[replete]

I've tested this yesterday and it's working well. I think there's a slight race condition somewhere though as sometimes a deployment following a webhook fails, but then redeploying again works. If I can reproduce this again I'll make a report, but it's definitely working as expected. Thanks

[BvdMerwe]

@andrasbacsai When I selected the new option to keep the files, I got the following error:

`[2024-Jul-19 22:18:51.092462] Starting deployment of splendid-sardine-sccww40 to localhost. [2024-Jul-19 22:18:51.337747] Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:latest. [2024-Jul-19 22:18:56.207443] ---------------------------------------- [2024-Jul-19 22:18:56.212857] Importing (commit sha HEAD) to /artifacts/lggswcw. [2024-Jul-19 22:19:05.633228] Pulling & building required images. [2024-Jul-19 22:19:06.004280] Removing old containers. [2024-Jul-19 22:19:06.353648] Starting new application. [2024-Jul-19 22:19:08.006655] rm:

[2024-Jul-19 22:19:08.015507] refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/.' rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/..'

[2024-Jul-19 22:19:08.419927] rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/.' rm: refusing to remove '.' or '..' directory: skipping '/data/coolify/applications/sccww40/..' `

I am also experiencing this. The container deploys as normal, but it reports that it failed.

[Desarso]

I get the same issue, most likely a scripting problem where it is trying to delete "." and ".." in the folder of the project. Also the docker "deploys" but it's really on a restart loop because the dockerfile cmd commands keep failing since it's failing to copy the repo files.

[replete]

Having real problems with this, race conditions and cleanup.

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/data/coolify/applications/xxxxxxx/nginx.prod.conf" to rootfs at "/etc/nginx/nginx.conf": mount /data/coolify/applications/xxxxxxx/nginx.prod.conf:/etc/nginx/nginx.conf (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type