search

coollabsio / coolify

An open-source & self-hostable Heroku / Netlify / Vercel alternative.
https://coolify.io
Apache License 2.0
34.62k stars 1.89k forks source link

[Bug]: DNS errors when cloning with GitHub API key #4318

Closed jarhodes314 closed 3 days ago

jarhodes314 commented 6 days ago

Error Message and Logs

The main error is fatal: unable to access 'https://github.com/myorg/repo.git/': Could not resolve host: github.com

2024-Nov-16 11:49:01.340379
docker run -d --network coolify --name jgs0g0kwk4wc84gckoccw4gk --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/coollabsio/coolify-helper:1.0.4
2024-Nov-16 11:49:01.340379
d8b0bf22d96ae4718a2f969414c0fdbee04a5224c7799dd994ebe7b72d4b3805
2024-Nov-16 11:49:06.162130
docker exec jgs0g0kwk4wc84gckoccw4gk bash -c 'GIT_SSH_COMMAND="ssh -o ConnectTimeout=30 -p 22 -o Port=22 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote https://x-access-token:%3CREDACTED%3E@github.com/myorg/repo.git trunk'
2024-Nov-16 11:49:06.162130
d94960e8c4ca17e5ca599fa2894751b50be49465    refs/heads/trunk
2024-Nov-16 11:49:06.534257
----------------------------------------
2024-Nov-16 11:49:06.542906
Importing myorg/repo:trunk (commit sha HEAD) to /artifacts/jgs0g0kwk4wc84gckoccw4gk.
2024-Nov-16 11:49:06.884216
docker exec jgs0g0kwk4wc84gckoccw4gk bash -c 'git clone -b "trunk" https://x-access-token:%3CREDACTED%3E@github.com/myorg/repo.git /artifacts/jgs0g0kwk4wc84gckoccw4gk && cd /artifacts/jgs0g0kwk4wc84gckoccw4gk && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git submodule update --init --recursive && cd /artifacts/jgs0g0kwk4wc84gckoccw4gk && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git lfs pull'
2024-Nov-16 11:49:06.884216
Cloning into '/artifacts/jgs0g0kwk4wc84gckoccw4gk'...
2024-Nov-16 11:49:18.966696
fatal: unable to access 'https://github.com/myorg/repo.git/': Could not resolve host: github.com
2024-Nov-16 11:49:19.097716
Oops something is not okay, are you okay? 😢
2024-Nov-16 11:49:19.108455
Cloning into '/artifacts/jgs0g0kwk4wc84gckoccw4gk'...
2024-Nov-16 11:49:19.108455
fatal: unable to access 'https://github.com/myorg/repo.git/': Could not resolve host: github.com
2024-Nov-16 11:49:19.118653
Deployment failed. Removing the new version of your application.

Steps to Reproduce

  1. Deploy an app from GitHub via "Private Repository (with GitHub app)" setting.

Example Repository URL

No response

Coolify Version

v4.0.0-beta.370

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 22.04

Additional Information

I'd presume it's something to do with #2568.

I have verified by reproducing manually the error is occurring specifically with the git clone. I can connect to github.com via HTTPS from both the host machine and containers running on it with curl

djsisson commented 6 days ago

@jarhodes314

if you run nslookup github.com what server is it using for name lookup

the build is running inside a container and can have a different dns resolver, which im guessing isnt working properly

also can you check 

journalctl -u docker.service -S "15 minutes ago"

its possible to set a dns resolver in /etc/docker/daemon.json you can add 


{
    dns: ["1.1.1.1","8.8.8.8"],
   ...
}
jarhodes314 commented 6 days ago

also can you check

journalctl -u docker.service -S "15 minutes ago"
Nov 16 15:10:39 orangepi5 dockerd[1486]: time="2024-11-16T15:10:39.954399833Z" level=warning msg="failed to close stdin: NotFound: task 8944193b3f9b2dbf2dc0ec5cad69e49f77bb366f825f61c83a1054538f46cf5d not found: not found"
Nov 16 15:11:07 orangepi5 dockerd[1486]: time="2024-11-16T15:11:07.211745017Z" level=info msg="ignoring event" container=5e31d0c5beeb160840c8e0b817949f234245b73e74e239309f5c24a2a6cc025d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 16 15:12:11 orangepi5 dockerd[1486]: time="2024-11-16T15:12:11.496145327Z" level=info msg="ignoring event" container=5e31d0c5beeb160840c8e0b817949f234245b73e74e239309f5c24a2a6cc025d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 16 15:13:02 orangepi5 dockerd[1486]: time="2024-11-16T15:13:02.702974845Z" level=info msg="ignoring event" container=b4f99d7bd9134541fc3c1bb31bff31ad2140b1d108d3e985d8fa62b7d8c5d6c7 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 16 15:13:03 orangepi5 dockerd[1486]: time="2024-11-16T15:13:03.003601954Z" level=error msg="error removing container" container=b4f99d7bd9134541fc3c1bb31bff31ad2140b1d108d3e985d8fa62b7d8c5d6c7 error="removal of container b4f99d7bd9134541fc3c1bb31bff31ad2140b1d108d3e985d8fa62b7d8c5d6c7 is already in progress"
Nov 16 15:13:15 orangepi5 dockerd[1486]: time="2024-11-16T15:13:15.709254554Z" level=info msg="ignoring event" container=5e31d0c5beeb160840c8e0b817949f234245b73e74e239309f5c24a2a6cc025d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 16 15:13:16 orangepi5 dockerd[1486]: time="2024-11-16T15:13:16.676568166Z" level=info msg="ignoring event" container=26140dc0aeae419a2f2b20d9db802604dc6b23d552889ddbf8fe823a80be8e27 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 16 15:13:16 orangepi5 dockerd[1486]: time="2024-11-16T15:13:16.716571496Z" level=warning msg="failed to close stdin: NotFound: task 26140dc0aeae419a2f2b20d9db802604dc6b23d552889ddbf8fe823a80be8e27 not found: not found"

From another container, including within the coolify network, I can copy the git command that coolify is running and it works as expected (i.e. it fails with an authentication error because I haven't supplied the token).

Additionally:

docker exec jgs0g0kwk4wc84gckoccw4gk bash -c 'GIT_SSH_COMMAND="ssh -o ConnectTimeout=30 -p 22 -o Port=22 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote https://x-access-token:<REDACTED>@github.com/myorg/repo.git trunk'
2024-Nov-16 11:49:06.162130
d94960e8c4ca17e5ca599fa2894751b50be49465    refs/heads/trunk
2024-Nov-16 11:49:06.534257

from the start of the logs indicates that the container is able to connect to github as it's retrieving the correct git hash. I am unable to reproduce the issue outside of the coolify build process.

jarhodes314 commented 3 days ago

I don't quite know why, but after downgrading to 367 and upgrading to 370, it appears the issue has magically gone away (after the upgrade, not before).