Kubernetes: Nginx (airbyte-webapp pod) cannot listen ipv6

[magefister]

## Environment - **Airbyte version**: app.kubernetes.io/version: 0.35.15-alpha, helm.sh/chart: airbyte-0.3.0 - **OS Version / Instance**: Kubernetes 1.15 - **Deployment**: Kubernetes deploy Helm - **Source Connector and version**: (if applicable example Salesforce 0.2.3)

• Destination Connector and version: (if applicable example Postgres 0.3.3)
• Severity: High
• Step where error happened: Deploy

Current Behavior

nginx cannot listen ipv6 so, Back-off restarting failed container, airbyte-webapp pod

Expected Behavior

no Back-off restarting failed container airbyte-webapp pod

Logs

If applicable, please upload the logs from the failing operation. For sync jobs, you can download the full logs from the UI by going to the sync attempt page and clicking the download logs button at the top right of the logs display window.

LOG

``` /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: ipv6 not available /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2022/02/04 03:15:34 [emerg] 1#1: socket() [::]:80 failed (97: Address family not supported by protocol) nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol) ```

Steps to Reproduce

git clone git@github.com:airbytehq/airbyte.git cd airbyte helm repo add bitnami https://charts.bitnami.com/bitnami helm dep build charts/airbyte helm upgrade --install \ --create-namespace \ --namespace "airbyte" \ --debug \ --wait \ --timeout "600s" \ contra-airbyte ./charts/airbyte

Are you willing to submit a PR?

Remove this with your answer.

[alafanechere]

Hi @magefister I think this is related to two things:

• the Nginx image used by the webapp listens on ipv6 by default
• Your Kubernetes cluster does not use ipv6 ? (maybe because your Kubernetes version is too old?)

This issue is more related to the nginx image than to Airbyte according to this one https://github.com/nginxinc/docker-nginx/issues/394. But https://github.com/nginxinc/docker-nginx/issues/394 is closed so maybe upgrading our Nginx image to a more recent version would fix this problem?

[dgagnon-simpleset]

airbyte does have ipv6 enabled in the default template however: https://github.com/airbytehq/airbyte/blob/master/airbyte-webapp/nginx/default.conf.template

[Startouf]

I am working with a software development company specialized in kubernetes deployments, they told me that for a long time, it was recommended to completely disable ipv6 when installing kubernetes. That it is less of a problem now, but that ipv6 it still generates problems occasionally and make things more complicated to setup (load balancer, dual network) and to protect (it increases the attack surface).

Our clusters have been deployed in ipv4 only and therefore we have the same problem. Could we make the ipv6 listening configurable ? I'd even suggest disabling it by default, it should not be too much of a breaking change ?

[Startouf]

Here is a modified Dockerfile that you can use to remove ipv6 listening

# Dockerfile
FROM airbyte/webapp:0.35.12-alpha

RUN rm /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
COPY remove_ipv6.sh /docker-entrypoint.d/10_remove_ipv6.sh
RUN chmod +x /docker-entrypoint.d/10_remove_ipv6.sh

And the script

# remove_ipv6.sh
#! /bin/sh
echo "Removing ipv6 from default nginx config"
sed -i 's/listen  \[::\]:80;/#/' /etc/nginx/templates/default.conf.template

In case someone wants to have a quick look, I have pushed it under myjobglasses/airbyte:latest. I'm not planning to maintain this, it's just to unlock my k8s setup so I can test airbyte.. So you can deploy the helm chart with the following values

webapp:
  image:
    repository: myjobglasses/airbyte
    tag: latest
    pullPolicy: Always

[qrpike]

This is another solution if you do not want to build your own docker image and maintain it:

In the webapp.yml file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: airbyte-webapp

(...)

          volumeMounts:
            - name: config
              mountPath: /etc/nginx/templates/
      volumes:
        - name: config
          configMap:
            name: airbyte-webapp-nginx
            items:
              - key: default.conf.template
                path: default.conf.template

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: airbyte-webapp-nginx
data:
  default.conf.template: |
    upstream api-server {
        server $INTERNAL_API_HOST;
    }

    upstream connector-builder-server {
      server $CONNECTOR_BUILDER_API_HOST;
    }

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;
        #access_log  /var/log/nginx/host.access.log  main;

        add_header Content-Security-Policy "script-src * 'unsafe-inline'; worker-src self blob:;";

        location / {
            root   /usr/share/nginx/html;
            try_files $uri $uri/ /index.html;

            sub_filter </head>
                    '</head><script language="javascript"\>
                    window.TRACKING_STRATEGY = "$TRACKING_STRATEGY";
                    window.AIRBYTE_VERSION = "$AIRBYTE_VERSION";
                    window.API_URL = "$API_URL";
                    window.CONNECTOR_BUILDER_API_URL = "$CONNECTOR_BUILDER_API_URL";
                    </script>';
            sub_filter_once on;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        location /api/ {
            fastcgi_read_timeout 1h;
            proxy_read_timeout 1h;
            client_max_body_size 200M;
            proxy_pass http://api-server/api/;
        }

        location /connector-builder-api/ {
            fastcgi_read_timeout 1h;
            proxy_read_timeout 1h;
            client_max_body_size 200M;
            proxy_pass http://connector-builder-server/;
        }
    }