pod exec not work - Githubissues

annProg commented 6 years ago

code

$k8sClient->setNamespace('dev');
$pods = $k8sClient->pods()->setLabelSelector(['app' => 'dev-router'])->find();

foreach($pods as $pod) {
    $arr = $pod->toArray();
    if($arr['status']['phase'] == 'Running') {
        print_r($k8sClient->pods()->exec($pod, ['cat','/etc/nginx/nginx.conf']));break;
    }
}

return 400 Bad Request with message Upgrade request required

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRe (truncated...)

samuelhautcoeur commented 5 years ago

Same issue here. The exec endpoint wants a websocket connection. This was handled nicely in godaddy/kubernetes-client: https://github.com/godaddy/kubernetes-client/issues/319

momon commented 5 years ago

I've just encountered this issue. Would this ever be addressed? I am trying to figure out how to get an exec command working via the API like this. I have the exact same error as above.

Message: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

maclof commented 5 years ago

I can take a look at this soon

What version of kubernetes are you running?

momon commented 5 years ago

@maclof that would be awesome. I am exploring alternatives but if this is fixed here that would make my life a lot easier right away.

My cluster is currently on 1.12.6-gke.11 but I do tend to keep up to date (still in development).

momon commented 5 years ago

@maclof have you been able to take a look at this?

Thanks!

maclof commented 5 years ago

I've finally had time to start working on this a bit as I have a need for it in my production environment.

I do have a semi working implementation of this, basically instead of sending a HTTP request to the /exec endpoint it now uses a new websocket client dependency to send this request.

momon commented 5 years ago

That's great to hear. Much appreciate the effort.

I look forward to that commit so I can try it out.

maclof commented 5 years ago

The latest version (v0.14.0) should have fixed this.

I did have some difficulty getting it working on my production Kubernetes v1.9.6 cluster, but I tested it with a v1.14.1 Kubernetes cluster and all is working OK

See below for an example:

$pod = $kubernetesClient->pods()->setFieldSelector([
    'metadata.name' => 'mongodb-0',
])->first();

$response = $kubernetesClient->pods()->exec($pod, [
    'command' => ['ps'],
    'stdout'  => true,
    'stderr'  => true,
]);

If the command you want to execute has multiple arguments, you would define these as seperate elements of the array, e.g.

$command = ['ls', '-al'];

The $response will contain an array of outputted messages, if the message length is 0 it will not be added to the array.

array:1 [▼
  0 => array:2 [▼
    "channel" => "stdout"
    "message" => """
        PID TTY          TIME CMD
          1 ?        01:47:03 mongod
       2208 ?        00:00:00 ps
      """
  ]
]

momon commented 5 years ago

Much appreciate this @maclof but seems not to be working for me on 1.13.7-gke.8 (which is the latest one available on GKE).

I seem unable to select my pod using:

$pod = $kubernetesClient->pods()->setFieldSelector([
    'metadata.name' => 'myname',
])->first();

It returns null.

I also tried with labelselector and find but it returns an empty array (as if the deployment had no pods) but they are there because I can fetch them via CLI and see them working. So I am unsure if the latest commit or perhaps the cluster version could be an issue here. I don't see a way to upgrade to 1.14.1 on GKE

momon commented 5 years ago

Ok I think I figured out the problem but unsure how to solve it. It appears composer does not want to update past 0.13.0 from your repo. When I try to set 0.14.0 I get this error:

composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package maclof/kubernetes-client 0.14.0 exists as maclof/kubernetes-client[0.0.1, 0.0.10, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.10.0, 0.11.0, 0.12.0, 0.12.1, 0.13.0, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, dev-master, dev-revert-23-master] but these are rejected by your constraint.

So as per above, it would appear that I can't upgrade to 0.14.0 ?? any clues?

Update: looks like this is the issue: https://packagist.org/packages/maclof/kubernetes-client packagist does not know about the 0.14.0 version.

momon commented 5 years ago

I was finally able to update to 0.14.0 by updating directly from VCS, however, I still can't get this working.

I am getting:

Fatal error: Call to a member function close() on null in /home/***/web/***/vendor/maclof/kubernetes-client/src/Client.php on line 478

Here is an example of my request:

                        $this->gke->setNamespace('default');
                        $pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();

                        foreach($pods as $pod) {
                            $arr = $pod->toArray();

                            if($arr['status']['phase'] == 'Running') {

                                $response = $this->gke->pods()->exec($pod, [
                                    'command' => ['ps'],
                                    'stdout'  => true,
                                    'stderr'  => true,
                                ]);

                            }
                        }

                        echo '<pre>';
                        print_r($response);
                        echo '</pre>';

maclof commented 5 years ago

I think my GitHub webhooks are out of date which is why you couldn’t update via composer without specifying the vcs url directly, I’ll get this fixed for the next release

I can fix your reported error about calling close on null, but that likely means it wasn’t able to connect for some reason.

What options are you specifying to the client for authentication? Client certificates / token etc? It’s been a while since I’ve used GKE

momon commented 5 years ago

@maclof on my sandbox (which is where I am testing this) I am using endpoint IP, username and password.

I am pretty sure it is connecting because I do get the data from the cluster on

$pods = $this->gke->pods()->setLabelSelector(['app' => 'myappname'])->find();

The error shows only on

$this->gke->pods()->exec($pod, [
                                    'command' => ['ps'],
                                    'stdout'  => true,
                                    'stderr'  => true,
                                ]);

maclof commented 5 years ago

@momon can you share the full schema for the pod? Remove any environment variable passwords, etc.

momon commented 5 years ago

@maclof here it is (I've removed / masked some sensitive info):

momon commented 5 years ago

@maclof how's it going? have you had a chance to look at this? Thanks!

momon commented 5 years ago

Hello @maclof just looking to check if you've had a change to look this over. Thanks!

maclof commented 5 years ago

@momon unfortunately not yet able to replicate on my clusters. Any chance of getting access to a cluster you’re working with? Perhaps setup a temporary gke cluster for a few days?

momon commented 5 years ago

@maclof happy to give you access to my sandbox, that's what it is intended for. Just unsure what details you would need exactly. Would the IP, usr and pwd for the actual cluster do so you can connect via the API or would you be needing me to add your service account to GKE?

maclof commented 5 years ago

@momon api access is perfect, if you could setup a demo pod that you’d try to use this with and I’ll get it all working

Drop the details via email to me at maclof@gmail.com

momon commented 5 years ago

@maclof sorry for the late response. I've sent you that email with the details.

Cheers, Jose R. Lopez

maclof commented 5 years ago

@momon please try version 0.14.1, I've got it working within that pod you emailed me as an example :)

momon commented 5 years ago

@maclof can confirm it to be working now on 0.14.3 (which is what composer upgrade upgraded me to).

Much appreciate your help with this, I can now finish off some parts of the app.

Cheers, Jose R. Lopez

maclof commented 5 years ago

@momon excellent news, let me know how you get on!

leonboot commented 4 years ago

I was very happy to see that version 0.14 introduced the ability to perform an exec on a pod, so I upgraded to the latest version (0.15.0 at the moment) and tried to execute a command. I'm having no luck, however :(

Running the following code results in an empty array being returned:

// This following line actually returns a Pod instance, not null
$pod = $this->client->pods()->setLabelSelector(['label' => 'somelabel'])->setFieldSelector(['status.phase' => 'Running'])->first();

$result = $client->pods()->exec($pod, [
    'command' => ['ls'],
    'container' => 'phpfpm',
    'stdout' => 'true',
    'stderr' => 'true',
]);

var_dump($result);

At first, I thought it might be because I'm talking to my cluster through a proxy (using kubectl proxy, with the --disable-filter flag). Then I created a service account, bound a role to it with enough privileges to perform a pod exec and tried again. Unfortunately the result was the same.

Are there any minimum requirements to the cluster that I'm talking to? In my case, it's a 1.12 GKE cluster.

maclof commented 4 years ago

@leonboot should work with GKE without issue, the cluster I tested was a GKE cluster.

If you can temporarily provide me access to the cluster and a code example to run, I'm sure I could get this working. If not you'll need to try and provide some more info to help debug what the problem is.

leonboot commented 4 years ago

@maclof I'm trying to create a test setup on our cluster with a service account which, annoyingly, works. I'm getting the expected output. So, on the bright side, the code works like a charm :) It's up to me now to figure out the difference between my live environment and the test setup. I'll share my findings if I think they might help others. Thanks for the input anyways!

maclof commented 4 years ago

@leonboot please share what the issue is if and when you find it, it could be something that we can put a check in place for to aid in debugging

leonboot commented 4 years ago

I've figured it out what the issue was: a trailing slash in the Kubernetes API endpoint URL 🤦‍♂

I used new Client(['master' => 'http://localhost:8001/']) to instantiate my client. While debugging, I added a var_dump($fullUrl) in the Client->sendUpgradeRequest() method, and noticed a double slash after the hostname (http://localhost:8001//api/v1/...) I removed the slash at the end of the endpoint URL and got the results I expected! 🎉

The trailing slash only seems to affect the React client. Normal requests all work as intended. So maybe it's worth to put some validation mechanism in place when performing an upgraded request?

meme commented 4 years ago

It seems that I can exec, but I'd like to interact with stdin to pipe files up to the node, like in kubectl cp. Is it possible to get a handle on this WebSocket connection so I can provide stdin input?

meme commented 4 years ago

Whipped something up (tar xf - -C /tmp/bar) using the code in the upgrade request function (assumes $data and $podName are in scope):

define("CHANNELS", [
    'stdin',
    'stdout',
    'stderr',
    'error',
    'resize'
]);

$connectorOptions = [
    'timeout' => 20,
    'tls' => [
        'cafile' => CA_CERT_PATH,
        'local_cert' => CLIENT_CERT_PATH,
        'local_pk' => CLIENT_KEY_PATH
    ]
];

$uri = 'wss://' . MASTER . '/api/v1/namespaces/default/pods/' . $podName . '/exec?command=tar&command=xf&command=-&command=-C&command=/tmp/bar&stdin=true&stderr=true&stdout=true';

$eventLoop = ReactFactory::create();
$connector = new WebSocketConnector($eventLoop, new ReactSocketConnector($eventLoop, $connectorOptions));
$connectionPromise = $connector($uri, ['channel.k8s.io'], []);

$socket = null;
$messages = [];

$connectionPromise->then(function ($connection) use ($data, &$socket, &$messages) {
    $socket = $connection;

    $connection->on('message', function ($message) use (&$messages) {
        /** @var Message $message */
        $data = $message->getPayload();

        $channel = CHANNELS[ord(substr($data, 0, 1))];
        $message = base64_decode(substr($data, 1));

        $messages[] = [
            'channel' => $channel,
            'message' => $message
        ];
    });

    $connection->send("\x00$data");
    $connection->close();
}, function ($e) {
    throw $e;
});

$eventLoop->run();

if ($socket) {
    $socket->close();
}

var_dump($messages);

Would be nice to have the ability to interact with the standard-in of the exec function, or perhaps a copy function for the pod object would be in order?

maclof commented 4 years ago

@meme apologies, for some reason I never saw your response to this.

Could you please open another issue for this? I'd love to get stdin support working correctly

meme commented 4 years ago

For sure: https://github.com/maclof/kubernetes-client/issues/66

maclof / kubernetes-client

pod exec not work #41