search

zonca / jupyterhub-deploy-kubernetes-jetstream

Configuration files for my tutorials on deploying JupyterHub on top of Kubernetes on XSEDE Jetstream (Openstack)
https://zonca.dev/categories/#jetstream
23 stars 14 forks source link

Cluster create error #21

Closed ktyle closed 5 years ago

ktyle commented 5 years ago

I am following the steps as in https://zonca.github.io/2019/06/kubernetes-jupyterhub-jetstream-magnum.html to deploy a Kubernetes cluster on my Jetstream/IU allocation.

When I run bash create_cluster.sh, after about 30-40 minutes, the task fails (see the "faults" field) with an authorization error that appears tied to the Docker container.

| Field               | Value                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                                                                                                                                                                                                                                               |
| cluster_template_id | 0c4bbb98-4e5d-4e26-b631-0e104ec1ec37                                                                                                                                                                                                                                                                                                                                                                                                        |
| node_addresses      | []                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| uuid                | b285f67b-e6ad-4f7c-8a64-c1f8642e2126                                                                                                                                                                                                                                                                                                                                                                                                        |
| stack_id            | ad218710-16b4-440f-8ce3-e54ae8dc7026                                                                                                                                                                                                                                                                                                                                                                                                        |
| status_reason       | Resource CREATE failed: Unauthorized: resources.kube_minions.resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)                                                                                                                                                                                                                                                                     |
| created_at          | 2019-08-21T21:39:41+00:00                                                                                                                                                                                                                                                                                                                                                                                                                   |
| updated_at          | 2019-08-21T22:39:42+00:00                                                                                                                                                                                                                                                                                                                                                                                                                   |
| coe_version         | v1.11.1                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| labels              | {'cloud-provider-enabled': 'true'}                                                                                                                                                                                                                                                                                                                                                                                                          |
| faults              | {'0': 'Unauthorized: resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)', 'kube_minions': 'Unauthorized: resources.kube_minions.resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)', 'docker_volume_attach': 'Unauthorized: resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)'} |
| keypair             | ktyle-api-key                                                                                                                                                                                                                                                                                                                                                                                                                               |
| api_address         | None                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| master_addresses    | []                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| create_timeout      | 60                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| node_count          | 1                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| discovery_url       | https://discovery.etcd.io/7957161c43212e9d076327ac2bde6341                                                                                                                                                                                                                                                                                                                                                                                  |
| master_count        | 1                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| container_version   | 1.12.6                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| name                | k8s                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| master_flavor_id    | m1.medium                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| flavor_id           | m1.medium                                                                                                                                                                                                                                                                                                                                                                                                                                   || Field               | Value                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                                                                                                                                                                                                                                               |
| cluster_template_id | 0c4bbb98-4e5d-4e26-b631-0e104ec1ec37                                                                                                                                                                                                                                                                                                                                                                                                        |
| node_addresses      | []                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| uuid                | b285f67b-e6ad-4f7c-8a64-c1f8642e2126                                                                                                                                                                                                                                                                                                                                                                                                        |
| stack_id            | ad218710-16b4-440f-8ce3-e54ae8dc7026                                                                                                                                                                                                                                                                                                                                                                                                        |
 **|** status_reason       | Resource CREATE failed: Unauthorized: resources.kube_minions.resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)**                                                                                                                                                                                                                                                                     |
| created_at          | 2019-08-21T21:39:41+00:00                                                                                                                                                                                                                                                                                                                                                                                                                   |
| updated_at          | 2019-08-21T22:39:42+00:00                                                                                                                                                                                                                                                                                                                                                                                                                   |
| coe_version         | v1.11.1                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| labels              | {'cloud-provider-enabled': 'true'}                                                                                                                                                                                                                                                                                                                                                                                                          |
| ****faults**              | {'0': 'Unauthorized: resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)', 'kube_minions': 'Unauthorized: resources.kube_minions.resources[0].resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)', 'docker_volume_attach': 'Unauthorized: resources.docker_volume_attach: The request you have made requires authentication. (HTTP 401)'} |**
| keypair             | ktyle-api-key                                                                                                                                                                                                                                                                                                                                                                                                                               |
| api_address         | None                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| master_addresses    | []                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| create_timeout      | 60                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| node_count          | 1                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| discovery_url       | https://discovery.etcd.io/7957161c43212e9d076327ac2bde6341                                                                                                                                                                                                                                                                                                                                                                                  |
| master_count        | 1                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| container_version   | 1.12.6                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| name                | k8s                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| master_flavor_id    | m1.medium                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| flavor_id           | m1.medium                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+---------------------+--------------------
+---------------------+--------------------

However, the two VM's get created and I can ssh into them: openstack server list

| 77e19f00-248a-4158-b927-6db16c25a1b1 | k8s-r27fvv6gy442-minion-0 | ACTIVE | ktyle-newk8s-network=10.0.0.21, 149.165.169.73                              | Fedora-AtomicHost-28-20180625 | m1.medium |
| 17016b34-56d4-45a3-88b1-f3de39b74101 | k8s-r27fvv6gy442-master-0 | ACTIVE | ktyle-newk8s-network=10.0.0.9, 149.165.169.49                               | Fedora-AtomicHost-28-20180625 | m1.medium |

Here is the output of create_template:

### +-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {'cloud-provider-enabled': 'true'}   |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | ktyle-newk8s-subnet1                 |
| master_flavor_id      | m1.medium                            |
| uuid                  | 0c4bbb98-4e5d-4e26-b631-0e104ec1ec37 |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | ktyle-api-key                        |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | 10                                   |
| server_type           | vm                                   |
| external_network_id   | public                               |
| cluster_distro        | fedora-atomic                        |
| image_id              | Fedora-AtomicHost-28-20180625        |
| volume_driver         | cinder                               |
| registry_enabled      | False                                |
| docker_storage_driver | devicemapper                         |
| apiserver_port        | -                                    |
| name                  | k8s_cluster_template                 |
| created_at            | 2019-08-21T21:36:38+00:00            |
| network_driver        | flannel                              |
| fixed_network         | ktyle-newk8s-network                 |
| coe                   | kubernetes                           |
| flavor_id             | m1.medium                            |
| master_lb_enabled     | False                                |
| dns_nameserver        | 8.8.8.8                              |
+-----------------------+--------------------------------------+
----------------------------------------------------------------------------------------------------------------

Wondering what needs to be done to resolve the authorization error.

--Kevin

zonca commented 5 years ago

@ktyle I created a cluster now and it completed successfully, I wonder if it was a temporary error. As you are doing testing on your current deployment, you could also launch another cluster (just change the name) and see if that succeeds.

ktyle commented 5 years ago

will do. In case it fails, I did do the following steps after installing kubectl on my local server that I'm running openstack on:

mkdir kubectl_secret cd kubectl_secret openstack coe cluster config k8s

But that returns "cluster in status CREATE_FAILED" so no config / certificate files get retrieved.

zonca commented 5 years ago

ok, so better remove the cluster and launch it again. We cannot easily get around that error pretending it worked.

zonca commented 5 years ago

If it fails again, try to connect as soon as possible to the master VM and look through:

kubectl get all --all-namespaces

to check if anything is failing and grab their logs.

ktyle commented 5 years ago

@zonca I think you were right that this was a temporary error! I have tried the cluster creation both yesterday evening and today and they succeeded. I will go ahead and proceed further with the deployment. Feel free to close.

zonca commented 5 years ago

great, thanks!