[Bug]: docker.host value in ~/.testcontainers.properties is not used

[giammin]

Testcontainers version

3.5.0

Using the latest Testcontainers version?

Yes

Host OS

windows

Host arch

x64

.NET version

7.0.403

Docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:22 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-buildx" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-buildx: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-compose" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-compose: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 19.38GiB
 Name: Nero
 ID: 46K3:5DBU:TFGT:UPFZ:36GJ:UCBL:5PGL:EGZT:VD3P:N7L4:YLFB:4SCV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: giammin
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

What happened?

I'm running testcontainers from a windows pc using the docker engine running on wsl2 (no docker desktop, custom docker installation directly on wsl2 debian)

It works fine if I set from code WithDockerEndpoint

 var _dbContainer = new MsSqlBuilder()
        .WithDockerEndpoint("tcp://localhost:2375")
        .WithImage("mcr.microsoft.com/mssql/server:2022-latest")
        .Build();

it work if i set environment var DOCKER_HOST

$env:DOCKER_HOST="tcp://localhost:2375" 
dotnet test

but if i set docker.host in ~/.testcontainers.properties

docker.host=tcp://localhost:1234 

I get the error:

Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured. You can customize your configuration using either the environment variables or the ~/.testcontainers.properties file. For more information, visit: https://dotnet.testcontainers.org/custom_configuration/ (Parameter 'DockerEndpointAuthConfig')

I know if it is reading that docker.host value because if I assign non valid url value it gives me an url exception:

System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Configurations.TestcontainersSettings' threw an exception. -------- System.InvalidOperationException : This operation is not supported for a relative URI.

Relevant log output

No response

Additional information

No response

[HofmeisterAn]

Hi, could you please verify that the environment variable has not been accidentally set to an empty string (please set it to $env:DOCKER_HOST=$null)? If TC can resolve the environment variable, it will favor it.

[giammin]

I did some test and I can confirm that $env:DOCKER_HOST is null

[HofmeisterAn]

Sorry, I cannot reproduce it. The properties file runs fine on my test host.

Your description confuses me. Since you are saying it picks up the property, I would expect that it really cannot connect to the Docker host. Is the port 1234 a typo? Your other two examples use the port 2375. I do not expect it to be an issue in Testcontainers.

[giammin]

the 1234 port was an error in a copy paste. My .testcontainers.properties file has the correct value:

docker.host=tcp://localhost:2375

how can i test what value is used in code?

could it be a MsSqlBuilder bug?

another thought: my .testcontainers.properties has only the docker.host line, do I need to set other variables?

[HofmeisterAn]

Testcontainers tries to resolve various Docker host configurations. It will select the first one it can establish a connection with. These are the configurations we check:

https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Configurations/TestcontainersSettings.cs#L29-L36

According to the error message, none of the providers is able to establish a connection to Docker. Debug into the following class to debug the environment or properties file configuration (it would be interesting which _dockerEngine it chooses):

https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Builders/EnvironmentEndpointAuthenticationProvider.cs#L28-L30

This part attempts to access (ping) the Docker host (if it succeeds, it will pick the endpoint and run the tests against it):

https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Builders/DockerEndpointAuthenticationProvider.cs#L36-L46

another thought: my .testcontainers.properties has only the docker.host line, do I need to set other variables?

No, that is not necessary. For example, Testcontainers Cloud does exactly the same to configure a Docker remote host.

[giammin]

I finally discovered what was the problem:

# comments in the .testcontainers.properties

I copy/pasted the example file from here: https://java.testcontainers.org/features/configuration/

and modified it so my docker.host file line was:

docker.host=tcp://my.docker.host:1234 # Equivalent to the DOCKER_HOST environment variable.

I deleted the comment and now it works as expected

[HofmeisterAn]

I deleted the comment and now it works as expected

😬

TBH, I am not sure if this is valid. Wikipedia says:

Comment lines in .properties files are denoted by the number sign (#) or the exclamation mark (!) as the first non blank character

But, of course, we can only select the value until the first # or ! occurs.

[giammin]

maybe the host value is not trimmed?

[HofmeisterAn]

Whitespaces are not a problem, but a non-blank characters following the host address are. The current implementation takes the value as it is. Given that .NET lacks native support for .properties files, we prioritize only the essential features.

https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Configurations/PropertiesFileConfiguration.cs#L43-L49

[giammin]

ok, so the error in my .testcontainers.properties was that the comment was not at the beginning of the line.

modifing the last line with this would prevent my error:

.ToDictionary(
     property => property[0], 
     property => property[1].Trim().Split(new[] { ' ','#' },2, StringSplitOptions.RemoveEmptyEntries)[0]);

[HofmeisterAn]

modifing the last line with this would prevent my error:

I had something similar in mind :+1:, but we need to take care of (consider) the escape sequences too.

I will close this issue in favor of #1041 (to keep the conversation clean). Thanks for bringing this up.

[giammin]

thanks for your support and for this awesome library