confluent-hub install through proxy

[z3rohour]

I'm trying to run docker-compose up -d --build according to documentation but the process gets stuck on building step:

Step 5/5 : RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
 ---> Running in c4b8de698b86
Running in a "--no-prompt" mode
java.net.ConnectException: Connection timed out (Connection timed out)

Server that I'm running docker on doesn't have direct Internet access and configured to use proxy server so I tried to add environment variables to Dockerfile:

ENV HTTP_PROXY="my_proxy"
ENV HTTPS_PROXY="my_proxy"

But it doesn't help.

I've pulled confluentinc/cp-kafka-connect:5.2.1 docker image, run it interactively and tried to install kafka-connect-datagen:latest manually

docker pull confluentinc/cp-kafka-connect:5.2.1
docker run -it --rm confluentinc/cp-kafka-connect:5.2.1 /bin/bash
export http_proxy=MY_PROXY_ADDRESS
export https_proxy=MY_PROXY_ADDRESS
confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest

But the image still cannot be downloaded.

curl -I https://api.hub.confluent.io/api/plugins says that connection was successful so obviously there is no proxy support for confluent-hub client. Is there any workaround for this issue?

[z3rohour]

As usual, I managed to fix the problem by myself right after posting :) Proxy parameters are taken from zulu-8-amd64 package so I simply add few lines into Dockerfile which define default proxy settings for Java:

RUN sed -i 's/# http.proxyHost=/http.proxyHost=MY_PROXY_ADDRESS/g' /usr/lib/jvm/zulu-8-amd64/jre/lib/net.properties && \
    sed -i 's/# http.proxyPort=80/http.proxyPort=MY_PROXY_PORT/g' /usr/lib/jvm/zulu-8-amd64/jre/lib/net.properties && \
    sed -i 's/# https.proxyHost=/https.proxyHost=MY_PROXY_ADDRESS/g' /usr/lib/jvm/zulu-8-amd64/jre/lib/net.properties && \
    sed -i 's/# https.proxyPort=443/https.proxyPort=MY_PROXY_PORT/g' /usr/lib/jvm/zulu-8-amd64/jre/lib/net.properties && \
    confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest

[rjmccabe3701]

@z3rohour what about authenticated proxies? Google says to add http{,s}_proxy{User,Password} but it doesn't seem to have any effect

[z3rohour]

@rjmccabe3701 not sure how to get it working in this case, I don't have proxy authentication enabled. You can try to set up system proxy settings instead of specifying them in net.properties configuration and then change java.net.useSystemProxy to 'true' in properties file but it didn't work out for me as far as I remember

[ybyzek]

@z3rohour -- can you please give this a try: https://hub.docker.com/r/cnfldemos/kafka-connect-datagen ? It's a pre-built Connect Docker image with kafka-connect-datagen plugin already installed

[jmgpeeters]

FWIW, setting the java.net.useSystemProxies=true did work for me. I'm also using zulu, but not inside docker.

[maurodoglio]

I solved this using JAVA_TOOL_OPTIONS to pass the https.proxyPort and https.proxyHost system variables

[vielfarbig]

I use the image confluentinc/ksqldb-server:0.17.0 and installation via confluent-hub install in the docker-compose command option, as you can see in https://github.com/confluentinc/demo-scene/blob/master/ksqldb-tombstones/docker-compose.yml. In this version zulu11 is used and you have to edit /usr/lib/jvm/zulu11/conf/net.properties. This cannot be edited by the docker-compose command option, because the user has not enough permissons. So I copied the net.properties with docker cp ksqldb:/usr/lib/jvm/zulu11/conf/net.properties ., edited and mounted the net.properties by:

    volumes:
      - ./net.properties:/usr/lib/jvm/zulu11/conf/net.properties

In the net.properties it is important to not use the schema, so http:// before the hosts, as so:

http.proxyHost=proxy.example.com
http.proxyPort=80
http.nonProxyHosts=localhost|127.*|[::1]

https.proxyHost=proxy.example.com
https.proxyPort=443