Add https_proxy support for kubeless builds behind firewalls

vallard commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?: Feature Request What happened: Running kubeless on a Kubernetes cluster doesn't take into account proxy settings required for build process to create runtime containers with correct dependencies. What you expected to happen: I hoped that adding in the https-proxy or https_proxy setting in serverless.yaml files would allow for build image to grab npm packages but it does not.

How to reproduce it (as minimally and precisely as possible): Would have to have a kubernetes cluster where pods could only reach the outside internet if an https proxy were added.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-10T11:44:36Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Kubeless version (use kubeless version):
```
Kubeless version: v1.0.0-alpha.8
```
Cloud provider or physical cluster: Bare metal kubernetes cluster installed with kubeadm.

andresmgot commented 5 years ago

Hi,

It should be possible to set those environment variables (http-proxy and https_proxy) for the container that install modules from the internet.

What's the output of executing kubectl get pod -o yaml -l function=<your-function>?

vallard commented 5 years ago

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cni.projectcalico.org/podIP: 192.168.4.40/32
      prometheus.io/path: /metrics
      prometheus.io/port: "8080"
      prometheus.io/scrape: "true"
    creationTimestamp: 2018-10-25T15:35:30Z
    generateName: list-585f746b99-
    labels:
      created-by: kubeless
      function: list
      pod-template-hash: "1419302655"
    name: list-585f746b99-6ff5j
    namespace: default
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: list-585f746b99
      uid: 9a9ea02c-d86b-11e8-beb5-0025b5888bdf
    resourceVersion: "4256578"
    selfLink: /api/v1/namespaces/default/pods/list-585f746b99-6ff5j
    uid: 9aa44900-d86b-11e8-beb5-0025b5888bdf
  spec:
    containers:
    - env:
      - name: FUNC_HANDLER
        value: list
      - name: MOD_NAME
        value: list
      - name: FUNC_TIMEOUT
        value: "180"
      - name: FUNC_RUNTIME
        value: nodejs6
      - name: FUNC_MEMORY_LIMIT
        value: "0"
      - name: FUNC_PORT
        value: "8080"
      - name: NODE_PATH
        value: /kubeless/node_modules
      image: kubeless/nodejs@sha256:f2a338c62d010687137c0880d1b68bea926f71a7111251a4622db8ae8c036898
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /healthz
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 3
        periodSeconds: 30
        successThreshold: 1
        timeoutSeconds: 1
      name: list
      ports:
      - containerPort: 8080
        protocol: TCP
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /kubeless
        name: list
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-qvkfw
        readOnly: true
    dnsPolicy: ClusterFirst
    initContainers:
    - args:
      - base64 -d < /src/list > /tmp/func.decoded && echo '3cf18b8bd678f903a42f36c2c6b18f6c2a0fd8cf4797b9d93c82fabae5095fd6  /tmp/func.decoded'
        > /tmp/func.sha256 && sha256sum -c /tmp/func.sha256 && unzip -o /tmp/func.decoded
        -d /kubeless && cp /src/package.json /kubeless
      command:
      - sh
      - -c
      image: kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
      imagePullPolicy: IfNotPresent
      name: prepare
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /kubeless
        name: list
      - mountPath: /src
        name: list-deps
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-qvkfw
        readOnly: true
    - args:
      - echo '7456c9194697fb8ff20a5a4347113c49e2cba8d7d1537dffc4d136189db0df20  /kubeless/package.json'
        > /tmp/deps.sha256 && sha256sum -c /tmp/deps.sha256 && npm config set registry
        https://registry.npmjs.org && npm install --production --prefix=/kubeless
      command:
      - sh
      - -c
      env:
      - name: HOME
        value: /tmp
      image: node:6.10
      imagePullPolicy: IfNotPresent
      name: install
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /kubeless
        name: list
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-qvkfw
        readOnly: true
      workingDir: /kubeless
    nodeName: kubec02
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 1000
      runAsUser: 1000
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - emptyDir: {}
      name: list
    - configMap:
        defaultMode: 420
        name: list
      name: list-deps
    - name: default-token-qvkfw
      secret:
        defaultMode: 420
        secretName: default-token-qvkfw
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2018-10-25T15:36:10Z
      message: 'containers with incomplete status: [install]'
      reason: ContainersNotInitialized
      status: "False"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: 2018-10-25T15:36:10Z
      message: 'containers with unready status: [list]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: 'containers with unready status: [list]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: 2018-10-25T15:35:30Z
      status: "True"
      type: PodScheduled
    containerStatuses:
    - image: kubeless/nodejs@sha256:f2a338c62d010687137c0880d1b68bea926f71a7111251a4622db8ae8c036898
      imageID: ""
      lastState: {}
      name: list
      ready: false
      restartCount: 0
      state:
        waiting:
          reason: PodInitializing
    hostIP: 172.28.225.140
    initContainerStatuses:
    - containerID: docker://d408388399cd0c7fd4ed23a6971d43118223acf9f3cd80886ec4f6cbc55ea562
      image: sha256:d2c5621fdfa3f219980e6d3149083b28b74e671ca8059fb1b7343750ae17b27d
      imageID: docker-pullable://kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
      lastState: {}
      name: prepare
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: docker://d408388399cd0c7fd4ed23a6971d43118223acf9f3cd80886ec4f6cbc55ea562
          exitCode: 0
          finishedAt: 2018-10-25T15:36:11Z
          reason: Completed
          startedAt: 2018-10-25T15:36:11Z
    - containerID: docker://2d745624620c953d255f841e211a71c67b22a50a2c0a35386c45c10a10a3f9d6
      image: node:6.10
      imageID: docker-pullable://node@sha256:39c92a576b42e5bee1b46bd283c7b260f8c364d8826ee07738f77ba74cc5d355
      lastState: {}
      name: install
      ready: false
      restartCount: 0
      state:
        running:
          startedAt: 2018-10-25T15:36:13Z
    phase: Pending
    podIP: 192.168.4.40
    qosClass: BestEffort
    startTime: 2018-10-25T15:36:10Z
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

vallard commented 5 years ago

I am able to make it work by editing the deployment:

kubectl edit deployment list

(where list is the name of my function) and then change the arguments to the init container by adding in my proxy setting with npm config set https-proxy http://myproxy. As an example it looks like:

> /tmp/deps.sha256 && sha256sum -c /tmp/deps.sha256 && npm config set https-proxy 
          http://proxy.example.com:80 && npm config set registry https://registry.npmjs.org
          && npm install --production --prefix=/kubeless

vallard commented 5 years ago

similarly in python I have to do this:

  - args:
        - echo 'cd3d3693961d04d5139f80ca9f53ed2044f628fa23a312d5aac1dd1a03a057f7  /kubeless/requirements.txt'
          > /tmp/deps.sha256 && sha256sum -c /tmp/deps.sha256 && https_proxy=myproxy:80
          pip install --prefix=/kubeless -r /kubeless/requirements.txt

vallard commented 5 years ago

and in golang do this:

    - args:
        - echo '144493f7cb03633804450b6eee3a40b30e0dd6b827986244070aa21129df97f0  /kubeless/Gopkg.toml'
          > /tmp/deps.sha256 && sha256sum -c /tmp/deps.sha256 && cd $GOPATH/src/kubeless
          && https_proxy=myproxy:80 dep ensure > /dev/termination-log
          2>&1

andresmgot commented 5 years ago

That shouldn't be necessary. Note that you can set environment variables with --env and those env vars will be set in the installation and the runtime containers. In your case, it should be enough to run kubeless function deploy --env https_proxy=myproxy:80 ...

vallard commented 5 years ago

Ahh... thank you! That did indeed seem to work:

kubeless function deploy list --env https_proxy=myproxy.com:80 --runtime python2.7 --from-file handler.py --handler handler.list --dependencies requirements.txt

I also found out I needed to change the serverless.yaml file to include the environment variables. Here is what it looks like:

service: guestbook

provider:
  name: kubeless
  hostname: 172.28.225.184.xip.io
  defaultDNSResolution: 'xip.io'
  runtime: python2.7

plugins:
  - serverless-kubeless

functions:
  create:
    handler: handler.create
    environment:
      https_proxy: myproxy.com:80
    events:
      - http:
          path: /create

  list:
    handler: handler.list
    environment:
      https_proxy: myproxy.com:80
    events:
      - http:
          path: /list

Before I was putting the proxy in the top under the provider variables. Putting in the actual functions seem to work better. Hopefully that helps someone! Thanks @andresmgot !

vmware-archive / kubeless

Add https_proxy support for kubeless builds behind firewalls #931