search

caicloud / ciao

Kernel for Kubeflow in Jupyter Notebook
Apache License 2.0
67 stars 18 forks source link

kernel can not started successly if kubeconfig is not set #81

Closed minshenglin closed 5 years ago

minshenglin commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

The ciao log shows Kubeflow kernel starts failed:

[KernelGatewayApp] Kernel args: {'kernel_name': u'kubeflow', 'env': {'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'KERNEL_LAUNCH_TIMEOUT': u'40', u'KERNEL_WORKING_DIR': u'/home/jovyan'}}
[I 190731 06:30:15 web:2162] 201 POST /api/kernels (10.233.80.43) 23.44ms
[I 190731 06:30:15 web:2162] 200 GET /api/kernels/4711b4c3-0f7e-4ddb-9650-1b96f31477d5 (10.233.80.43) 1.05ms
2019/07/31 06:30:15 Using config file: /etc/ciao/config.yaml
ERROR: logging before flag.Parse: W0731 06:30:15.738126       9 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: W0731 06:30:15.738144       9 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
2019/07/31 06:30:15 Error building kubeConfig: invalid configuration: no configuration has been provided
[KernelGatewayApp] Initializing websocket connection /api/kernels/4711b4c3-0f7e-4ddb-9650-1b96f31477d5/channels
[KernelGatewayApp] WARNING | No session ID specified
[KernelGatewayApp] Requesting kernel info from 4711b4c3-0f7e-4ddb-9650-1b96f31477d5
[KernelGatewayApp] Connecting to: tcp://127.0.0.1:35719
[KernelGatewayApp] KernelRestarter: restarting kernel (1/5), new random ports
[KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-4711b4c3-0f7e-4ddb-9650-1b96f31477d5.json']
[KernelGatewayApp] Connecting to: tcp://127.0.0.1:46113
2019/07/31 06:30:18 Using config file: /etc/ciao/config.yaml
ERROR: logging before flag.Parse: W0731 06:30:18.709932      19 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: W0731 06:30:18.709942      19 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
2019/07/31 06:30:18 Error building kubeConfig: invalid configuration: no configuration has been provided
[KernelGatewayApp] KernelRestarter: restarting kernel (2/5), new random ports
[KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-4711b4c3-0f7e-4ddb-9650-1b96f31477d5.json']
[KernelGatewayApp] Connecting to: tcp://127.0.0.1:34603
2019/07/31 06:30:21 Using config file: /etc/ciao/config.yaml
ERROR: logging before flag.Parse: W0731 06:30:21.722328      25 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: W0731 06:30:21.722337      25 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
2019/07/31 06:30:21 Error building kubeConfig: invalid configuration: no configuration has been provided
[KernelGatewayApp] KernelRestarter: restarting kernel (3/5), new random ports
[KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-4711b4c3-0f7e-4ddb-9650-1b96f31477d5.json']
[KernelGatewayApp] Connecting to: tcp://127.0.0.1:38448
2019/07/31 06:30:24 Using config file: /etc/ciao/config.yaml
ERROR: logging before flag.Parse: W0731 06:30:24.730290      32 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: W0731 06:30:24.730299      32 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
2019/07/31 06:30:24 Error building kubeConfig: invalid configuration: no configuration has been provided

What you expected to happen:

The kernel gateway prevents the kernel to inherited environment variables when spawning, so KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT are empty.

See: https://github.com/jupyter/kernel_gateway/pull/280

How to reproduce it (as minimally and precisely as possible):

Using the following ./hack/k8s.config.yaml to build docker image:

namespace: ciao
s2i:
  provider: configmap

Anything else we need to know?: