Open hathvi opened 1 week ago
I probably should have just created a PR for this and we could discuss this further there. Let me know if you wish for me to do so and I'll find some time.
I'm not sure if the approach here is correct, since the DNS entry for accessing the host should now work for all recent docker versions for example:
package testcontainers_test
import (
"bytes"
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/docker/docker/api/types/container"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)
const caddyFileContent = `
listen :80
reverse_proxy /api/* {
to {$API_SERVER}
health_uri /health
health_status 200
health_interval 10s
}
`
func TestCaddyfile(t *testing.T) {
ctx := context.Background()
apiServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, World!")
}))
t.Cleanup(apiServer.Close)
u, err := url.Parse(apiServer.URL)
require.NoError(t, err)
_, port, err := net.SplitHostPort(u.Host)
require.NoError(t, err)
u.Host = "host.docker.internal:" + port
caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: testcontainers.ContainerRequest{
Image: "caddy:2.8.4",
ExposedPorts: []string{"80/tcp"},
WaitingFor: wait.ForLog("server running"),
Env: map[string]string{
"API_SERVER": u.String(),
},
Files: []testcontainers.ContainerFile{
{
Reader: bytes.NewReader([]byte(caddyFileContent)),
ContainerFilePath: "/etc/caddy/Caddyfile",
},
},
HostConfigModifier: func(hc *container.HostConfig) {
hc.ExtraHosts = append(hc.ExtraHosts, "host.docker.internal:host-gateway")
},
},
Started: true,
})
testcontainers.CleanupContainer(t, caddyContainer)
require.NoError(t, err)
caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http")
require.NoError(t, err)
resp, err := http.Get(caddyURL + "/api/test")
require.NoError(t, err)
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
require.Equal(t, "Hello, World!\n", string(body))
lr, err := caddyContainer.Logs(ctx)
require.NoError(t, err)
lb, err := io.ReadAll(lr)
require.NoError(t, err)
fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb))
}
Thanks @stevenh!
I like this approach better for my use case but I'll keep this issue and PR open and let the project maintainers decide if it's worth pulling in? Given your solution, I'm not sure what the use case is for HostAccessPorts
and if my change would make a difference?
Thank you for your insight!
One of the benefits of the HostAccessPorts
is that it can expose ports listening on 127.0.0.1
which is the case of the httptest.NewServer
which was originally why I was swapping out the httptest.Server.Listener
. In my example this was just junk left over from trying to come up with a solution, but listening on 0.0.0.0
wasn't technically needed with my fix. In your example you still need to swap out the Listener
as talking to host-gateway
talks to the bridge interface which would require the Listener
to listen on all interfaces or for me to determine the bridge interface address to listen on.
One of the benefits of the
HostAccessPorts
is that it can expose ports listening on127.0.0.1
which is the case of thehttptest.NewServer
which was originally why I was swapping out thehttptest.Server.Listener
. In my example this was just junk left over from trying to come up with a solution, but listening on0.0.0.0
wasn't technically needed with my fix. In your example you still need to swap out theListener
as talking tohost-gateway
talks to the bridge interface which would require theListener
to listen on all interfaces or for me to determine the bridge interface address to listen on.
I'm not sure I understand, httptest.NewServer
in the example above listens on 127.0.0.1
and works fine. Could you clarify what you believe HostAccessPorts
solves?
Thanks @stevenh!
I like this approach better for my use case but I'll keep this issue and PR open and let the project maintainers decide if it's worth pulling in? Given your solution, I'm not sure what the use case is for
HostAccessPorts
and if my change would make a difference?
As one of the community maintainers, happy to consider your PR, I'm just trying to understand the use case, and if its really needed anymore?
I'm wondering if HostAccessPorts
pre-dates the internal DNS entries working on all platforms, as I know it was flaky for some time on Windows as an example. @mdelapenya do you know the history on this?
I just came across this issue yesterday as I was exploring a similar problem with HostAccessPorts
and the SSH tunnel not being set up in time - I too was depending on the host network being accessible during container startup.
The DNS solution works great and is certainly simpler, so thanks for that @stevenh !
My $.02 --
IMO, if the support for HostAccessPorts
is to be kept around in spite of this, then I do think the fix provided by @hathvi is a logical one - I had implemented a similar custom container lifecycle hook to get things working the way we needed before coming across this issue.
If indeed there is no justification for continuing to support HostAccessPorts
other than backwards compatibility, then it would be great to see the docs updated to point out the availability of host.docker.internal
and perhaps even to illustrate its use with an example like the one here.
Thanks for your thoughts @jeremyg484, can anyone think of a situation which HostAccessPorts
can handle which the DNS route can't?
Are you two by chance on Mac or Windows? I think Windows, under Docker Desktop, routes traffic to the host a bit differently then Mac or Linux. I use Linux as my primary OS and am not super familiar with Windows. The test that @stevenh provided does not work for me, which I'll describe why below.
I'd preface that my original problem with testcontainers was that it was non obvious how to access the host using testcontainers. I however was also not aware of the docker extra hosts config and my searching around testcontainers lead me to HostAccessPorts
which I found to be setting up the listeners to late.
Docker by default uses a bridge network. It creates a new interface on the host named docker0
with it's own subnet, in my case this is 172.17.0.1/16
with the interface address being 172.17.0.1
. When you configure extra hosts to host-gateway
it creates a DNS record in /etc/hosts to host.docker.internal 172.17.0.1
.
When you create a new httptest.Server
it by default binds to 127.0.0.1
on a random port of the machine you are running the app on, there's no direct way of changing this behavior and you must swap out the httptest.Server.Listener
to bind to a different interface. In order for the DNS solution to work the httptest.Server
would need to bind to 172.17.0.1
( the container gateway / bridge interface ).
HostAccessPorts
requests the SSH server bind the remote port in the container to 127.0.0.1:PORT
and tunnel traffic back to the client which in turn dials 127.0.0.1:PORT
( the httptest.Server
) on the client machine and copies those bytes from the tunnel to the server. If I were to swap out the httptest.Server.Listener
to bind to 172.17.0.0.1
I'd have the same issue with HostAccessPorts
but DNS would work.
My guess would be that you guys are likely on Windows and/or running Docker Desktop which likely does something similar to what HostAccessPorts
is doing and/or Windows allows connecting to a port bound on 127.0.0.1
from any interface. I don't have a Windows VM currently to do any testing.
I would say HostAccessPorts
does solve a problem, especially since people are generally going to be using this for testing and I assume httptest.Server
would be a somewhat common use case for people testing HTTP clients, however I don't think this is a problem testcontainers necessarily needs to resolve as it feels out of scope, it's just helpful that it does.
I would also say that if HostAccessPorts
were kept it might be nice to extend its functionality to change which interface the local client dials, working more inline with how the ssh -R
works. e.g., ssh -R [local-addr:]local-port:remote-addr:remote-port destination
. Right now testcontainers doesn't expose the ability to set local-addr
or remote-addr
and hard codes these to localhost
( here for the remote listener, and here for the local forwarding client ). I assume this would result in a breaking change however as you'd likely need something like HostAccessPorts []string
instead of []int
and support values like local-addr:port/tcp
, port/tcp
( defaults to 127.0.0.1 ), local-addr:port
( defaults to tcp ), port
( defaults to 127.0.0.1 and tcp )
I don't contribute as much to open source as I should mainly due to anxiety socializing with others, but I'd be down to contributing the suggested changes if that seems like a good idea to you guys.
Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to HostAccessPorts
really make sense as this would only be a problem for servers bound to 127.0.0.1
. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.
Thanks for all the extra info. I wonder what the expected behaviour should be. Could you confirm what the name resolves to?
Oh to confirm my test host was Windows under WSL. So would be interested to understand that OSs work and what doesn't
Tested on Mac and it also works fine.
Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to
HostAccessPorts
really make sense as this would only be a problem for servers bound to127.0.0.1
. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.
Thats still a legitimate problem to solve.
Thanks for all the extra info. I wonder what the expected behaviour should be. Could you confirm what the name resolves to?
If we're referring to DNS using the extra hosts set to host-gateway
then it's expected that an entry in /etc/hosts is added that points to the docker0 interface, which is what I get.
$ docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.1 host.docker.internal
172.17.0.2 7c7f24b879a3
If I listen on 127.0.0.1
on the host and try curling from a container using the DNS solution then I would not expect to be able to as the server is not listening on that interface, which is what I get.
$ nc -l 127.0.0.1 8888
$ docker run --rm -it --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (7) Failed to connect to host.docker.internal port 8888 after 0 ms: Could not connect to server
If I listen on 172.17.0.1
then I would expect to be able to access the server from the container using the DNS solution as it's listening on that interface, which is what I get.
$ nc -l 172.17.0.1 8888
HEAD / HTTP/1.1
Host: host.docker.internal:8888
User-Agent: curl/8.10.1
Accept: */*
^C
$ docker run --rm -it --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (52) Empty reply from server
Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to
HostAccessPorts
really make sense as this would only be a problem for servers bound to127.0.0.1
. If the server was bound to any other interface other then a loopback then the server should be accessible to the container as long as the host has a route to it.Thats still a legitimate problem to solve.
To clarify, I was only referring to my suggested changes on adding the ability to specify the local address to connect to in HostAccessPorts
, I believe that suggestion was redundant.
It should only be a problem if the server you're trying to talk to is bound to an address on the host that is not routable from the container. This is the case for 127.0.0.1 as the container also creates an loopback interface with an addr of 127.0.0.1, so talking to 127.0.0.1 would just send to the container loopback interface and not the host.
For example I can connect to my tailscale interface from a container just fine as the container doesn't have a route for this address and sends the request out the default gateway to docker0 on the host, and the host has a route to the tailscale address so sends it out that interface.
$ ip -4 addr show tailscale0
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
inet 100.78.128.77/32 scope global tailscale0
valid_lft forever preferred_lft forever
$ nc -l 100.78.128.77 8888
HEAD / HTTP/1.1
Host: 100.78.128.77:8888
User-Agent: curl/8.10.1
Accept: */*
^C
$ docker run --rm -it curlimages/curl curl --head http://100.78.128.77:8888
curl: (52) Empty reply from server
I think there's a legitimate helpful use case for HostAccessPorts
but only for the following cases
httptest.NewServer
)docker0
to bind to that address or have tests break if ran with a different network mode.I'm not sure how/why this is working on windows and macos unless binding a port on 127.0.0.1 for some results in it listening on all interfaces. On your Mac are you also using Docker Desktop? I still speculate that Docker Desktop is doing something to expose the host to the VM that results in ports bound on 127.0.0.1 working from the container.
What does host.docker.internal
resolve to in your container? Only Linux, regardless of which network mode I use, it's always set to the docker0 IP of the host. However, if I set the network mode to host
I can talk to ports bound to 127.0.0.1 on the host but only if the request is to 127.0.0.1 in the container and not host.docker.internal. I wonder if possibly host.docker.internal:host-gateway
on windows is setting the address to 127.0.0.1
in the container and you have docker desktop set to use a host network by default?
$ docker run --rm -it --network host --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://host.docker.internal:8888
curl: (7) Failed to connect to host.docker.internal port 8888 after 0 ms: Could not connect to server
$ docker run --rm -it --network host --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://127.0.0.1:8888
curl: (52) Empty reply from server
$ docker run --rm -it --network bridge --add-host host.docker.internal:host-gateway curlimages/curl curl --head http://127.0.0.1:8888
curl: (7) Failed to connect to 127.0.0.1 port 8888 after 0 ms: Could not connect to server
To add some further examples of usage to the discussion - I have not tested on Windows, but my specific usage of host.docker.internal
does work on both Mac (our dev machines, using Docker Desktop) and Ubuntu (our CI runners).
I am setting ExtraHosts
to host.docker.internal:host-gateway
just as in the example from @stevenh above. I then have our app running in the container set to use host.docker.internal
as a substituted address specifically for this testing scenario, replacing the actual address of an external 3rd-party service for which we're providing a test double using the httptest
package.
I am setting a custom net.Listener
on the httptest.Server
, because our scenario requires talking to specific predefined ports instead of the random open port that gets used by default in httptest.NewUnstartedServer
. Note (and I would think this relates to what @hathvi has explored above) that I am creating the listener with
l, err := net.Listen("tcp", ":"+port)
according to the docs for net.Listen
, "For TCP networks, if the host in the address parameter is empty or a literal unspecified IP address, Listen listens on all available unicast and anycast IP addresses of the local system".
I just checked, and if I specify the loopback address 127.0.0.1
explicitly in net.Listen
, it still works on my Mac, but fails in our Ubuntu CI runner.
That's what I see too, Linux seems to be the odd one out.
I wouldn't call Linux the odd one out, it's working as it should, I think Mac, Windows or something else is just configured slightly different within these test environments and those are the odd ones out. Docker is Linux native, and Docker for me is working just as I would expect given the defaults the Docker documentation define and how networking in general works both inside and outside containers.
Can you run a few commands @stevenh?
docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
so we can see if it's getting set to the docker0 gateway address like it does on Linuxdocker inspect CONTAINER_NAME/ID -f '{{json .NetworkSettings.Networks}}'
so we can see what network mode is being used. Since you're not explicitly setting it in your test, it should in theory be bridge according to the docs, but I suspect it's host in your case.netstat -a -n
or you can open the resource manager and see all listeners there.docker version
- I'm running 27.3.1 for referenceI can set up a Windows VM later tonight ~8PM PT w/ Docker Desktop and WSL, just so I can explore more mainly out of curiosity now. I think given the differences we see just in this thread gives HostAccessPorts
a use. If you can talk to a server on the host bound to 127.0.0.1 from a container then great, if not you can use HostAccessPorts
as a solution. We just need to resolve the late listening issue this issue is for. I think there's value in continuing to explore and potentially updating the docs to reflect our findings to better guide people in the future so they don't try hacking their own solutions.
docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
so we can see if it's getting set to the docker0 gateway address like it does on Linuxdocker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 192.168.65.254 host.docker.internal 172.17.0.2 32db41e7dd87
- Block the test before terminate and run
docker inspect CONTAINER_NAME/ID -f '{{json .NetworkSettings.Networks}}'
so we can see what network mode is being used. Since you're not explicitly setting it in your test, it should in theory be bridge according to the docs, but I suspect it's host in your case.docker inspect 020989b419e6 -f '{{json .NetworkSettings.Networks}}' {"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:03","DriverOpts":null,"NetworkID":"f4e593f77e57e8d759b34578d47ca89583c3cae50f4ac18b5cf9d447a97cebb2","EndpointID":"0dcb292a563a3041263672990cce0c49efcb28b60f3eca4e6d3614d82ac7d31b","Gateway":"172.17.0.1","IPAddress":"172.17.0.3","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}
- It would also be interesting to see what address the port is being bound to on windows, which looks like you can see with netstat -a -n or you can open the resource manager and see all listeners there.
netstat -na |grep 45757 |grep LIST tcp 30 0 127.0.0.1:45757 0.0.0.0:* LISTEN
docker version - I'm running 27.3.1 for reference
docker version Client: Version: 27.2.0 API version: 1.47 Go version: go1.21.13 Git commit: 3ab4256 Built: Tue Aug 27 14:14:20 2024 OS/Arch: linux/amd64 Context: default
Server: Docker Desktop () Engine: Version: 27.2.0 API version: 1.47 (minimum version 1.24) Go version: go1.21.13 Git commit: 3ab5c7d Built: Tue Aug 27 14:15:15 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.20 GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353 runc: Version: 1.1.13 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0
> I wouldn't call Linux the odd one out, it's working as it should, I think Mac, Windows or something else is just configured slightly different within these test environments and those are the odd ones out. Docker is Linux native, and Docker for me is working just as I would expect given the defaults the Docker documentation define and how networking in general works both inside and outside containers.
We shouldn't assume that because docker is native on Linux that the intent wasn't that seen on Windows or Mac with regards being able to connect to loopback address. It could be a bug with either, as we should be able to rely on consistent behaviour across all platforms.
I was not able to get WSL2 running within VirtualBox last night, but I got things set up on a laptop this AM and can confirm your test works for me on Windows as well.
Based on the outputs you provided in your last post this should technically not be allowed unless something in the middle is doing something with the network packets such as NAT or proxying to the host. It looks like WSL2 uses NAT by default, but doesn't allow this behavior itself.
PS C:\Users\justin\workspace\test> go run server.go
Listening: 127.0.0.1:51403
PS C:\Users\justin\workspace\test> wsl -d ubuntu
justin@jlap:/mnt/c/Users/justin/workspace/test$ ip route
default via 172.26.192.1 dev eth0 proto kernel
172.26.192.0/20 dev eth0 proto kernel scope link src 172.26.196.73
justin@jlap:/mnt/c/Users/justin/workspace/test$ curl -m 5 172.26.192.1:51403
curl: (28) Connection timed out after 5002 milliseconds
The WSL2 documentation even makes explicit mention of this https://learn.microsoft.com/en-us/windows/wsl/networking#connecting-via-remote-ip-addresses
When using remote IP addresses to connect to your applications, they will be treated as connections from the Local Area Network (LAN). This means that you will need to make sure your application can accept LAN connections.
For example, you may need to bind your application to 0.0.0.0 instead of 127.0.0.1. In the example of a Python app using Flask, this can be done with the command: app.run(host='0.0.0.0'). Keep security in mind when making these changes as this will allow connections from your LAN.
I tried digging more into how Docker is running under WSL2 but at first glance doesn't seem as straight forward as I was hoping. I'll try digging more into this later tonight.
Testcontainers version
0.33.0
Using the latest Testcontainers version?
Yes
Host OS
Linux
Host arch
x86_64
Go version
1.23.1
Docker version
Client: Docker Engine - Community Version: 27.3.1 API version: 1.47 Go version: go1.22.7 Git commit: ce12230 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Context: default
Server: Docker Engine - Community Engine: Version: 27.3.1 API version: 1.47 (minimum version 1.24) Go version: go1.22.7 Git commit: 41ca978 Built: Fri Sep 20 11:41:00 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.22 GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc: Version: 1.1.14 GitCommit: v1.1.14-0-g2c9f560 docker-init: Version: 0.19.0 GitCommit: de40ad0
Docker info
Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose
Server: Containers: 14 Running: 9 Paused: 0 Stopped: 5 Images: 34 Server Version: 27.3.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-40-generic Operating System: Ubuntu 22.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 62.52GiB Name: jhome ID: 43fdd48e-011e-40da-aff1-b76bc378d203 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
What happened?
I'm attempting to utilize testcontainers-go to test my Caddy configuration as a gateway to my API server but I'm running into problems with how testcontainers-go exposes host ports and I believe this issue to be a bug.
Setup
In my tests, I've set up a
httptest.Server
to act as my API server, listening on a random port on the host. I then set up Caddy in a testcontainer and expose the API server port to the container viaHostAccessPorts
. My Caddy configuration defines the API server with a health check which Caddy checks on startup.caddyfile_test.go
```go package caddy_test import ( "bytes" "context" "fmt" "io" "net" "net/http" "net/http/httptest" "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/testcontainers/testcontainers-go" "github.com/testcontainers/testcontainers-go/wait" ) const caddyFileContent = ` listen :80 reverse_proxy /api/* { to {$API_SERVER} health_uri /health health_status 200 health_interval 10s } ` func TestCaddyfile(t *testing.T) { ctx := context.Background() apiServerListener, err := net.Listen("tcp", "0.0.0.0:0") assert.NoError(t, err) apiServerPort := apiServerListener.Addr().(*net.TCPAddr).Port apiServer := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "Hello, World!") })) apiServer.Listener.Close() apiServer.Listener = apiServerListener apiServer.Start() caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ ContainerRequest: testcontainers.ContainerRequest{ Image: "caddy:2.8.4", ExposedPorts: []string{"80/tcp"}, WaitingFor: wait.ForLog("server running"), Env: map[string]string{ "API_SERVER": fmt.Sprintf("http://%s:%d", testcontainers.HostInternal, apiServerPort), }, Files: []testcontainers.ContainerFile{ { Reader: bytes.NewReader([]byte(caddyFileContent)), ContainerFilePath: "/etc/caddy/Caddyfile", }, }, HostAccessPorts: []int{apiServerPort}, }, Started: true, }) require.NoError(t, err) defer caddyContainer.Terminate(ctx) caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http") require.NoError(t, err) resp, err := http.Get(caddyURL + "/api/test") require.NoError(t, err) defer resp.Body.Close() body, err := io.ReadAll(resp.Body) require.NoError(t, err) assert.Equal(t, http.StatusOK, resp.StatusCode) assert.Equal(t, "Hello, World!\n", string(body)) lr, err := caddyContainer.Logs(ctx) assert.NoError(t, err) lb, err := io.ReadAll(lr) assert.NoError(t, err) fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb)) } ```Test Output
```diff == Caddy Logs == {"level":"info","ts":1727952070.1965187,"msg":"using config from file","file":"/etc/caddy/Caddyfile"} {"level":"info","ts":1727952070.1969736,"msg":"adapted config to JSON","adapter":"caddyfile"} {"level":"warn","ts":1727952070.1969776,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1} {"level":"info","ts":1727952070.1972885,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]} {"level":"warn","ts":1727952070.1973321,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80} {"level":"info","ts":1727952070.1973393,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1727952070.1973433,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"} {"level":"info","ts":1727952070.1973994,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000736a80"} {"level":"info","ts":1727952070.1974878,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"} {"level":"info","ts":1727952070.197532,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."} {"level":"info","ts":1727952070.1975832,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]} +{"level":"info","ts":1727952070.1976013,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]} {"level":"info","ts":1727952070.1976032,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["listen"]} -{"level":"info","ts":1727952070.1976056,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"} -{"level":"info","ts":1727952070.1976073,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"} {"level":"info","ts":1727952070.1978004,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"} {"level":"info","ts":1727952070.1978037,"msg":"serving initial configuration"} {"level":"info","ts":1727952070.197835,"logger":"tls.obtain","msg":"acquiring lock","identifier":"listen"} {"level":"info","ts":1727952070.1985145,"logger":"tls.obtain","msg":"lock acquired","identifier":"listen"} {"level":"info","ts":1727952070.1985347,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"listen"} {"level":"info","ts":1727952070.1985307,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"} {"level":"info","ts":1727952070.1986136,"logger":"tls","msg":"finished cleaning storage units"} -{"level":"error","ts":1727952070.3384068,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_ip":"172.17.0.1","remote_port":"54434","client_ip":"172.17.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:33500","uri":"/api/test","headers":{"User-Agent":["Go-http-client/1.1"],"Accept-Encoding":["gzip"]}},"duration":0.000040091,"status":503,"err_id":"qe8hu1acn","err_trace":"reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:486)"} ================ --- FAIL: TestCaddyfile (1.10s) caddyfile_test.go:76: Error Trace: /home/justin/workspace/test/caddyfile_test.go:76 Error: Not equal: expected: 200 actual : 503 Test: TestCaddyfile caddyfile_test.go:77: Error Trace: /home/justin/workspace/test/caddyfile_test.go:77 Error: Not equal: expected: "Hello, World!\n" actual : "" Diff: --- Expected +++ Actual @@ -1,2 +1 @@ -Hello, World! Test: TestCaddyfile FAIL FAIL github.com/hathvi/test 1.189s FAIL ```Problem
My problem with this set up is that Caddy logs a "connection refused" error for the health check even though the testcontainer is ready. I attempt to make a request to the Caddy server after startup but receive an HTTP 502 Bad Gateway error as the API server wasn't initially reachable even though it's running and accepting connections on the host. Caddy will continue to return an HTTP 502 until the next health check.
My Analysis
I can see that
HostAccessPorts
utilizes a separate container running an SSH server then sets up aPostReadies
lifecycle hook on the Caddy container in order to then set up the forwarding in the SSH container. It appears to do the forwarding by firing off a go routine that connects to the SSH container with remote port forwarding, listening to theHostAccessPorts
ports on the SSH container and tunneling this to the host.PostReadies
seems like a lot to late to setup the forwarding. I'm utilizingHostAccessPorts
so I can talk to a server on the host from my testcontainer, so in order for my test container to be ready I would expect to be able to talk to that server before I do any of my testing. Logically I would assume I should be able to have a wait strategy that depends on that connection being made.Test fix
I created a fork and updated
exposeHostPorts
to setup a lifecycle hook onPreCreates
instead ofPostReadies
. This ensures the host port is accessible via the SSH container to the testcontainer from all lifecycle hooks and container command.In theory this shouldn't break anything even if someone sets up the listener for the host port in a later lifecycle hook as connections back on the host port are only established once you try connecting to the remote port.
testcontainers-go.patch
```diff diff --git a/port_forwarding.go b/port_forwarding.go index 88f14f2d..ad17fb10 100644 --- a/port_forwarding.go +++ b/port_forwarding.go @@ -150,8 +150,8 @@ func exposeHostPorts(ctx context.Context, req *ContainerRequest, ports ...int) ( // after the container is ready, create the SSH tunnel // for each exposed port from the host. sshdConnectHook = ContainerLifecycleHooks{ - PostReadies: []ContainerHook{ - func(ctx context.Context, c Container) error { + PreCreates: []ContainerRequestHook{ + func(ctx context.Context, req ContainerRequest) error { return sshdContainer.exposeHostPort(ctx, req.HostAccessPorts...) }, }, ```Relevant log output
No response
Additional information
No response