broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
990 stars 358 forks source link

java.net.SocketTimeoutException to establish connection with quay.io/50.17.122.58:443 behind a proxy #7136

Open bioprojects opened 1 year ago

bioprojects commented 1 year ago

Hello, I executed the following workflow using chromwell behind a proxy

$ java \ -Djava.net.useSystemProxies=true \ -Dhttp.proxyHost=202.241.78.237 -Dhttp.proxyPort=8080 \ -Dhttps.proxyHost=202.241.78.237 -Dhttps.proxyPort=8080 \ -jar cromwell-85.jar run public_health_bacterial_genomics/workflows/wf_theiaprok_illumina_pe.wdl -i input.json

but it caused the following error, indicating connection with quay.io/50.17.122.58:443 timed out.

2023-05-11 10:01:42,43] [info] Request threw an exception on attempt #1. Retrying after 596 milliseconds
org.http4s.client.ConnectionFailure: Error connecting to https://quay.io using address quay.io:443 (unresolved: false)
        at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144)
        at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139)
        at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119)
        at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119)
        at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83)
        at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172)
Caused by: java.net.SocketTimeoutException: An attempt to establish connection with quay.io/50.17.122.58:443 timed out after 10 seconds.
        at org.http4s.blaze.channel.nio2.ClientChannelFactory$$anon$1.run(ClientChannelFactory.scala:66)
        at org.http4s.blaze.util.Execution$$anon$3.execute(Execution.scala:80)
        at org.http4s.blaze.util.TickWheelExecutor$Node.run(TickWheelExecutor.scala:271)
        at org.http4s.blaze.util.TickWheelExecutor$Bucket.checkNext$1(TickWheelExecutor.scala:207)
        at org.http4s.blaze.util.TickWheelExecutor$Bucket.prune(TickWheelExecutor.scala:213)
        at org.http4s.blaze.util.TickWheelExecutor.go$3(TickWheelExecutor.scala:168)
        at org.http4s.blaze.util.TickWheelExecutor.org$http4s$blaze$util$TickWheelExecutor$$cycle(TickWheelExecutor.scala:171)
        at org.http4s.blaze.util.TickWheelExecutor$$anon$1.run(TickWheelExecutor.scala:68)

To confirm https (port 443) access to the quay.io/50.17.122.58 in this environment, I executed wget $ wget https://50.17.122.58/ but it caused

Resolving ... 202.241.78.237
Connecting to 202.241.78.237|:8080... connected.

ERROR: certificate common name ‘*.quay.io’ doesn't match requested host name ‘50.17.122.58’.
To connect to 50.17.122.58 insecurely, use `--no-check-certificate'.

I guess an option similar to `--no-check-certificate' in wget need to be implemented in cromwell to fix this problem. Could you please do it?

Many thanks in advance.

Koji Yahara Group Leader Antimicrobial Resistance Research Center National Institute of Infectious Diseases Japan

aednichols commented 1 year ago

The HTTP library we use [0] does not support proxies [1], therefore it is not possible for Cromwell to support them either without a whole-library replacement.

The certificate error is normal and a red herring, it occurs because certs apply to domain names and not IP addresses. I can reproduce it locally with no proxy.

[0] https://github.com/broadinstitute/cromwell/blob/17efd599d541a096dc5704991daeaefdd794fefd/project/Dependencies.scala#L166 [1] https://github.com/http4s/blaze/issues/656