Closed annProg closed 5 years ago
Same issue here. The exec endpoint wants a websocket connection. This was handled nicely in godaddy/kubernetes-client: https://github.com/godaddy/kubernetes-client/issues/319
I've just encountered this issue. Would this ever be addressed? I am trying to figure out how to get an exec command working via the API like this. I have the exact same error as above.
Message: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}
I can take a look at this soon
What version of kubernetes are you running?
@maclof that would be awesome. I am exploring alternatives but if this is fixed here that would make my life a lot easier right away.
My cluster is currently on 1.12.6-gke.11 but I do tend to keep up to date (still in development).
@maclof have you been able to take a look at this?
Thanks!
I've finally had time to start working on this a bit as I have a need for it in my production environment.
I do have a semi working implementation of this, basically instead of sending a HTTP request to the /exec endpoint it now uses a new websocket client dependency to send this request.
That's great to hear. Much appreciate the effort.
I look forward to that commit so I can try it out.
The latest version (v0.14.0) should have fixed this.
I did have some difficulty getting it working on my production Kubernetes v1.9.6 cluster, but I tested it with a v1.14.1 Kubernetes cluster and all is working OK
See below for an example:
$pod = $kubernetesClient->pods()->setFieldSelector([
'metadata.name' => 'mongodb-0',
])->first();
$response = $kubernetesClient->pods()->exec($pod, [
'command' => ['ps'],
'stdout' => true,
'stderr' => true,
]);
If the command you want to execute has multiple arguments, you would define these as seperate elements of the array, e.g.
$command = ['ls', '-al'];
The $response will contain an array of outputted messages, if the message length is 0 it will not be added to the array.
array:1 [▼
0 => array:2 [▼
"channel" => "stdout"
"message" => """
PID TTY TIME CMD
1 ? 01:47:03 mongod
2208 ? 00:00:00 ps
"""
]
]
Much appreciate this @maclof but seems not to be working for me on 1.13.7-gke.8 (which is the latest one available on GKE).
I seem unable to select my pod using:
$pod = $kubernetesClient->pods()->setFieldSelector([
'metadata.name' => 'myname',
])->first();
It returns null.
I also tried with labelselector and find but it returns an empty array (as if the deployment had no pods) but they are there because I can fetch them via CLI and see them working. So I am unsure if the latest commit or perhaps the cluster version could be an issue here. I don't see a way to upgrade to 1.14.1 on GKE
Ok I think I figured out the problem but unsure how to solve it. It appears composer does not want to update past 0.13.0 from your repo. When I try to set 0.14.0 I get this error:
composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- The requested package maclof/kubernetes-client 0.14.0 exists as maclof/kubernetes-client[0.0.1, 0.0.10, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.10.0, 0.11.0, 0.12.0, 0.12.1, 0.13.0, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, dev-master, dev-revert-23-master] but these are rejected by your constraint.
So as per above, it would appear that I can't upgrade to 0.14.0 ?? any clues?
Update: looks like this is the issue: https://packagist.org/packages/maclof/kubernetes-client packagist does not know about the 0.14.0 version.
I was finally able to update to 0.14.0 by updating directly from VCS, however, I still can't get this working.
I am getting:
Fatal error: Call to a member function close() on null in /home/***/web/***/vendor/maclof/kubernetes-client/src/Client.php on line 478
Here is an example of my request:
$this->gke->setNamespace('default');
$pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();
foreach($pods as $pod) {
$arr = $pod->toArray();
if($arr['status']['phase'] == 'Running') {
$response = $this->gke->pods()->exec($pod, [
'command' => ['ps'],
'stdout' => true,
'stderr' => true,
]);
}
}
echo '<pre>';
print_r($response);
echo '</pre>';
I think my GitHub webhooks are out of date which is why you couldn’t update via composer without specifying the vcs url directly, I’ll get this fixed for the next release
I can fix your reported error about calling close on null, but that likely means it wasn’t able to connect for some reason.
What options are you specifying to the client for authentication? Client certificates / token etc? It’s been a while since I’ve used GKE
@maclof on my sandbox (which is where I am testing this) I am using endpoint IP, username and password.
I am pretty sure it is connecting because I do get the data from the cluster on
$pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();
The error shows only on
$this->gke->pods()->exec($pod, [
'command' => ['ps'],
'stdout' => true,
'stderr' => true,
]);
@momon can you share the full schema for the pod? Remove any environment variable passwords, etc.
@maclof here it is (I've removed / masked some sensitive info):
Maclof\Kubernetes\Collections\PodCollection Object
(
[items:protected] => Array
(
[0] => Maclof\Kubernetes\Models\Pod Object
(
[schema:protected] => Array
(
)
[apiVersion:protected] => v1
[pluralKind:protected] =>
[attributes:protected] => Array
(
[metadata] => Array
(
[name] => dpl-fp-211-wp-4kjbc-7484b9b7bf-m89gr
[generateName] => dpl-fp-211-wp-4kjbc-7484b9b7bf-
[namespace] => default
[selfLink] => /api/v1/namespaces/default/pods/dpl-fp-211-wp-4kjbc-7484b9b7bf-m89gr
[uid] => 961cc092-a0dd-11e9-a181-42010a96016f
[resourceVersion] => 29089652
[creationTimestamp] => 2019-07-07T17:35:17Z
[labels] => Array
(
[app] => cust10-wp50
[pod-template-hash] => 7484b9b7bf
)
[annotations] => Array
(
[kubernetes.io/limit-ranger] => LimitRanger plugin set: cpu request for container ***
)
[ownerReferences] => Array
(
[0] => Array
(
[apiVersion] => apps/v1
[kind] => ReplicaSet
[name] => dpl-fp-211-wp-4kjbc-7484b9b7bf
[uid] => 109f0511-9ea2-11e9-a181-42010a96016f
[controller] => 1
[blockOwnerDeletion] => 1
)
)
)
[spec] => Array
(
[volumes] => Array
(
[0] => Array
(
[name] => pvc-fp-211-wp-4kjbc
[persistentVolumeClaim] => Array
(
[claimName] => pvc-fp-211-wp-4kjbc
)
)
[1] => Array
(
[name] => default-token-m62gt
[secret] => Array
(
[secretName] => default-token-m62gt
[defaultMode] => 420
)
)
)
[containers] => Array
(
[0] => Array
(
[name] => ***
[image] => ***
[ports] => Array
(
[0] => Array
(
[name] => ***
[containerPort] => 80
[protocol] => TCP
)
)
[env] => Array
(
***
)
[resources] => Array
(
[requests] => Array
(
[cpu] => 100m
)
)
[volumeMounts] => Array
(
[0] => Array
(
[name] => pvc-fp-211-wp-4kjbc
[mountPath] => /***
)
[1] => Array
(
[name] => default-token-m62gt
[readOnly] => 1
[mountPath] => /var/run/secrets/kubernetes.io/serviceaccount
)
)
[terminationMessagePath] => /dev/termination-log
[terminationMessagePolicy] => File
[imagePullPolicy] => Always
)
)
[restartPolicy] => Always
[terminationGracePeriodSeconds] => 30
[dnsPolicy] => ClusterFirst
[serviceAccountName] => default
[serviceAccount] => default
[nodeName] => gke-***-default-pool-aa24e883-jbz8
[securityContext] => Array
(
)
[schedulerName] => default-scheduler
[tolerations] => Array
(
[0] => Array
(
[key] => node.kubernetes.io/not-ready
[operator] => Exists
[effect] => NoExecute
[tolerationSeconds] => 300
)
[1] => Array
(
[key] => node.kubernetes.io/unreachable
[operator] => Exists
[effect] => NoExecute
[tolerationSeconds] => 300
)
)
[priority] => 0
[enableServiceLinks] => 1
)
[status] => Array
(
[phase] => Running
[conditions] => Array
(
[0] => Array
(
[type] => Initialized
[status] => True
[lastProbeTime] =>
[lastTransitionTime] => 2019-07-07T17:35:46Z
)
[1] => Array
(
[type] => Ready
[status] => True
[lastProbeTime] =>
[lastTransitionTime] => 2019-07-07T17:36:09Z
)
[2] => Array
(
[type] => ContainersReady
[status] => True
[lastProbeTime] =>
[lastTransitionTime] => 2019-07-07T17:36:09Z
)
[3] => Array
(
[type] => PodScheduled
[status] => True
[lastProbeTime] =>
[lastTransitionTime] => 2019-07-07T17:35:45Z
)
)
[hostIP] => 10.150.15.215
[podIP] => 10.4.12.21
[startTime] => 2019-07-07T17:35:46Z
[containerStatuses] => Array
(
[0] => Array
(
[name] => ***
[state] => Array
(
[running] => Array
(
[startedAt] => 2019-07-07T17:36:09Z
)
)
[lastState] => Array
(
)
[ready] => 1
[restartCount] => 0
[image] => ***
[imageID] => ***
[containerID] => ***
)
)
[qosClass] => Burstable
)
)
)
)
)
@maclof how's it going? have you had a chance to look at this? Thanks!
Hello @maclof just looking to check if you've had a change to look this over. Thanks!
@momon unfortunately not yet able to replicate on my clusters. Any chance of getting access to a cluster you’re working with? Perhaps setup a temporary gke cluster for a few days?
@maclof happy to give you access to my sandbox, that's what it is intended for. Just unsure what details you would need exactly. Would the IP, usr and pwd for the actual cluster do so you can connect via the API or would you be needing me to add your service account to GKE?
@momon api access is perfect, if you could setup a demo pod that you’d try to use this with and I’ll get it all working
Drop the details via email to me at maclof@gmail.com
@maclof sorry for the late response. I've sent you that email with the details.
Cheers, Jose R. Lopez
@momon please try version 0.14.1, I've got it working within that pod you emailed me as an example :)
@maclof can confirm it to be working now on 0.14.3 (which is what composer upgrade upgraded me to).
Much appreciate your help with this, I can now finish off some parts of the app.
Cheers, Jose R. Lopez
@momon excellent news, let me know how you get on!
I was very happy to see that version 0.14 introduced the ability to perform an exec on a pod, so I upgraded to the latest version (0.15.0 at the moment) and tried to execute a command. I'm having no luck, however :(
Running the following code results in an empty array being returned:
// This following line actually returns a Pod instance, not null
$pod = $this->client->pods()->setLabelSelector(['label' => 'somelabel'])->setFieldSelector(['status.phase' => 'Running'])->first();
$result = $client->pods()->exec($pod, [
'command' => ['ls'],
'container' => 'phpfpm',
'stdout' => 'true',
'stderr' => 'true',
]);
var_dump($result);
At first, I thought it might be because I'm talking to my cluster through a proxy (using kubectl proxy
, with the --disable-filter
flag). Then I created a service account, bound a role to it with enough privileges to perform a pod exec and tried again. Unfortunately the result was the same.
Are there any minimum requirements to the cluster that I'm talking to? In my case, it's a 1.12 GKE cluster.
@leonboot should work with GKE without issue, the cluster I tested was a GKE cluster.
If you can temporarily provide me access to the cluster and a code example to run, I'm sure I could get this working. If not you'll need to try and provide some more info to help debug what the problem is.
@maclof I'm trying to create a test setup on our cluster with a service account which, annoyingly, works. I'm getting the expected output. So, on the bright side, the code works like a charm :) It's up to me now to figure out the difference between my live environment and the test setup. I'll share my findings if I think they might help others. Thanks for the input anyways!
@leonboot please share what the issue is if and when you find it, it could be something that we can put a check in place for to aid in debugging
I've figured it out what the issue was: a trailing slash in the Kubernetes API endpoint URL 🤦♂
I used new Client(['master' => 'http://localhost:8001/'])
to instantiate my client. While debugging, I added a var_dump($fullUrl)
in the Client->sendUpgradeRequest()
method, and noticed a double slash after the hostname (http://localhost:8001//api/v1/...) I removed the slash at the end of the endpoint URL and got the results I expected! 🎉
The trailing slash only seems to affect the React client. Normal requests all work as intended. So maybe it's worth to put some validation mechanism in place when performing an upgraded request?
It seems that I can exec
, but I'd like to interact with stdin
to pipe files up to the node, like in kubectl cp
. Is it possible to get a handle on this WebSocket connection so I can provide stdin
input?
Whipped something up (tar xf - -C /tmp/bar
) using the code in the upgrade request function (assumes $data
and $podName
are in scope):
define("CHANNELS", [
'stdin',
'stdout',
'stderr',
'error',
'resize'
]);
$connectorOptions = [
'timeout' => 20,
'tls' => [
'cafile' => CA_CERT_PATH,
'local_cert' => CLIENT_CERT_PATH,
'local_pk' => CLIENT_KEY_PATH
]
];
$uri = 'wss://' . MASTER . '/api/v1/namespaces/default/pods/' . $podName . '/exec?command=tar&command=xf&command=-&command=-C&command=/tmp/bar&stdin=true&stderr=true&stdout=true';
$eventLoop = ReactFactory::create();
$connector = new WebSocketConnector($eventLoop, new ReactSocketConnector($eventLoop, $connectorOptions));
$connectionPromise = $connector($uri, ['channel.k8s.io'], []);
$socket = null;
$messages = [];
$connectionPromise->then(function ($connection) use ($data, &$socket, &$messages) {
$socket = $connection;
$connection->on('message', function ($message) use (&$messages) {
/** @var Message $message */
$data = $message->getPayload();
$channel = CHANNELS[ord(substr($data, 0, 1))];
$message = base64_decode(substr($data, 1));
$messages[] = [
'channel' => $channel,
'message' => $message
];
});
$connection->send("\x00$data");
$connection->close();
}, function ($e) {
throw $e;
});
$eventLoop->run();
if ($socket) {
$socket->close();
}
var_dump($messages);
Would be nice to have the ability to interact with the standard-in of the exec
function, or perhaps a copy
function for the pod object would be in order?
@meme apologies, for some reason I never saw your response to this.
Could you please open another issue for this? I'd love to get stdin support working correctly
code
return 400 Bad Request with message
Upgrade request required