Connecting to cluster without discovery external IP isn't working

[ghost]

It tries to connect to server via it's internal IPs which is not available on remote clusters

[astefanutti]

Thanks for the report.

It may be related to the attempt to update the server URL with the federation information, that is still incomplete: https://github.com/astefanutti/kubebox/blob/04f97f91ac3fdde06a03fb8a60af713096f8dd3d/lib/kubebox.js#L113

Would you be able to provide more information, for example, the result of a GET on the api resource and the serverAddressByClientCIDRs object?

[ghost]

From federation you mean multi cluster feature of kubernetes? We have normal clusters and they're not federated. If you try to connect to remote bare metal cluster you might see the error. I get this: connect ECONNREFUSED 192.168.2.50:6433 192.168.2.50 is IP of local network of cluster. It shouldn't try to send request to it.

Can you provide me instructions on how to provide more info?

[ghost]

Generally Kubebox should simply work the way kubectl works. I've copied binary to kubernetes master node and ran it from there and it's asking me for credentials while kubectl works fluently

[astefanutti]

Can you provide me instructions on how to provide more info?

you can do:

$ curl -k <MASTER_URL>/api

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.64.12:8443"
    }
  ]
}

[astefanutti]

Generally Kubebox should simply work the way kubectl works. I've copied binary to kubernetes master node and ran it from there and it's asking me for credentials while kubectl works fluently

Agreed. Kubebox currently sources client connection configuration from $HOME/.kube and default to using the current context. We need to cover all the possible cases. Would you mind creating a separate issue for this?

[ghost]

This is the result

~$ curl localhost:8080/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.2.50:6443"
    }
  ]
}

kubebox shouldn't use serverAddress, it should use config provided in cluster.server of kubeconfig

[astefanutti]

Taken from the Kubernetes REST API documentation:

https://github.com/kubernetes/kubernetes/blob/4a01f44b7378fb0c58450898e1bca30ea8a5158a/api/swagger-spec/storage.k8s.io.json#L73

a map of client CIDR to server address that is serving this group. This is to help clients reach servers in the most network-efficient way possible. Clients can use the appropriate server address as per the CIDR that they match. In case of multiple matches, clients should use the longest matching CIDR. The server returns only those CIDRs that it thinks that the client can match. For example: the master will return an internal IP CIDR only, if the client reaches the server using an internal IP. Server looks at X-Forwarded-For header or X-Real-Ip header or request.RemoteAddr (in that order) to get the client IP.

So it's surprising it returns on internal IP. For some reasons, it thinks it reachable from the client.

It seems you are proxying the master API from the curl command. Are you when running Kubebox?

[ghost]

It returns internal IP because it doesn't have external IP, Api server is behind a firewall. I connected to VM via ssh and ran curl command but I didn't do this when running kubebox. I think using cluster.server makes more sense that using serverAddress because with cluster.server you already have a connection to API server that even kubectl uses, but with serverAddress you might not

[astefanutti]

I understand your reasoning. Having had a quick look at it, it seems kubectl doesn't rely on discovery information. So I'll follow that logic and remove the discovery out.

[astefanutti]

Fixed in fabf5a202e1318fa4f2fc63ae8330bf24ae79071.

[astefanutti]

@k1-hedayati I've updated the binaries in 0.1.1 release if you want to give it a spin.

[ghost]

@astefanutti thank you, it's working now. By the way I don't see resource usage data even though I've installed heapster and I can use top pods command

[astefanutti]

@astefanutti thank you, it's working now.

Great. Thx for the feedback.

By the way I don't see resource usage data even though I've installed heapster and I can use top pods command

Currently Kubebox accesses the embedded cAdvisor, either from the Kubelet stats endpoint or directly, to get the data. Would you mind creating a separate issue with more details so that we can troubleshoot? (it's not like the first time I ask 😉)