chectl: che.infra.kubernetes.namespace usage & how to create different namespace for each user

[svkr2k]

• I started with che codebase tagged with v7.5.1. I set the value of che.infra.kubernetes.namespace.allow_user_defined=true in the file assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties.
• che.infra.kubernetes.namespace.default=<username>-che
• Created the che-server image
• Added the image to registry (ECR).
• Installed che using the following (used the che-server image from local ECR registry ): chectl server:start --platform=k8s --installer=helm --domain=mydomain.com --multiuser --tls --templates=./deploy --cheimage=<link-to-registry-image>/che-server:latest

Then i created a workspace, I could see that the newly created workspace still gets listed under 'che' namespace ?

~/src/che-7.5.1$ sudo kubectl get pod --namespace che
NAME                                                          READY   STATUS              RESTARTS   AGE
che-6dc7f5bdb6-rjq7g                                          1/1     Running             0          11m
devfile-registry-66ccfbd48c-bt72s                             1/1     Running             0          11m
keycloak-8798c4c79-hlzpt                                      1/1     Running             0          11m
plugin-registry-678667956c-j6kqr                              1/1     Running             0          11m
postgres-565dc5956b-fsw9q                                     1/1     Running             0          11m
workspacerszr1uror3fbdtkv.che-jwtproxy-864cc8bc9c-ft4mw       1/1     Running             0          4m
workspacerszr1uror3fbdtkv.che-workspace-pod-dc6b78674-kx4vt   2/2     Running             0          4m

When i create a second workspace, the following error appears:

Error: Failed to run the workspace: "Failure executing: POST at: https://56.63.0.1/api/v1/namespaces/che/services. Message: services "my-rest-server" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=services, name=my-rest-server, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=services "my-rest-server" already exists, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={})."

I tested the same for NodeJS 10 devfile also, this time the error message shown when 2nd workspace created was related to volume mounting error.

[svkr2k]

@sleshchenko @ibuziuk , @lkrejci, please help. Added more info.

[sparkoo]

@svkr2k editing che.properties don't have any effect as the value is overriden by helm chart https://github.com/eclipse/che/blob/master/deploy/kubernetes/helm/che/values.yaml#L65, but it should be default <username>-che so there is something wrong.

• Is the cheWorkspacesNamespace in your deploy/kubernetes/helm/che/values.yaml correct?
• can you please check che deployment env variables for CHE_INFRA_KUBERNETES_NAMESPACE and CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT
• please check che configmap for same variables

[sparkoo]

I've reproduced it locally. The value set in configmap is che.

[~/dev/che] (b9256eb439 ✘)✹✭ λ kubectl describe cm che -n che | grep -A 2 NAMESPACE
CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT:
----
che
--
CHE_INFRA_KUBERNETES_NAMESPACE:
----
che

If I run helm command directly, it is correct. My bet is on chectl.

[sparkoo]

this would be the issue https://github.com/che-incubator/chectl/blob/master/src/tasks/installers/helm.ts#L326

[sparkoo]

@svkr2k yes, you can either

• deploy directly with helm
• deploy with chectl and update configmap vars
• deploy with chectl and add deployment env vars to container

The doc what vars to set is here https://github.com/eclipse/che-docs/blob/master/src/main/pages/che-7/installation-guide/proc_configuring-namespace-strategies.adoc. You want One namespace per user strategy. And please be aware that CHE_INFRA_KUBERNETES_NAMESPACE is legacy variable and should be unset.

Let me know if you need further help with setting this.

[sparkoo]

@svkr2k looks like you're missing one _, it should be CHE_INFRA_KUBERNETES_PVC_ACCESS__MODE

[svkr2k]

Thank you @sparkoo, @tolusha . Here are the latest settings:

In che.properties:

che.infra.kubernetes.namespace=
che.infra.kubernetes.namespace.default=<username>-che
che.infra.kubernetes.namespace.allow_user_defined=true
che.infra.kubernetes.pvc.strategy=common
che.infra.kubernetes.pvc.access_mode=ReadWriteOnce
che.infra.kubernetes.pvc.wait_bound=true

In che-7.5.1/deploy/kubernetes/helm/che/values.yaml

global:
  cheWorkspacesNamespace: "<username>-che"
  cheWorkspacesPVCStrategy: "common"

In che-7.5.1/deploy/kubernetes/helm/che/templates/configmap.yaml:

  CHE_INFRA_KUBERNETES_NAMESPACE: ""
  CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: {{ .Values.global.cheWorkspacesNamespace | quote}}
  CHE_INFRA_KUBERNETES_PVC_STRATEGY: "{{ .Values.global.cheWorkspacesPVCStrategy }}"

Install che:

helm upgrade --install che --force --namespace che \
  --set global.cheDomain=mydomain.com \
  --set global.cheWorkspacesNamespace='<username>-che' \
  --set global.ingressDomain=mydomain.com \
  --set cheimage=<myregistry>/testorg/che-server:latest \
  -f /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/values/tls.yaml \
  -f /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/values/multi-user.yaml \
  /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/

$ kubectl describe cm che -n che | grep -A 2 NAMESPACE
CHE_INFRA_KUBERNETES_NAMESPACE:
----

CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT:
----
<username>-che

In browser, after I login, i try to create my first workspace. The following is the log with errors shown:

AttachVolume.Attach succeeded for volume "pvc-5bf1584a-2217-11ea-acba-029ec662a75d" 
Container image "eclipse/che-init-plugin-broker:v0.24" already present on machine
Created container
Started container
Cleaning /plugins dir
Starting Init Plugin Broker
Container image "eclipse/che-unified-plugin-broker:v0.24" already present on machine
Created container
Started container
Unified Che Plugin Broker
List of plugins and editors to install
- eclipse/my-rest-server-plugin/0.0.1 - Debug container.
- cherxp/che-ide/7.4.0 - IDE contianer
Starting Che plugins and editor processing
Starting VS Code and Theia plugins processing
All plugins have been successfully processed
Error: Failed to run the workspace: "Failure executing: POST at: https://100.64.0.1/api/v1/namespaces/r1-che/services. Message: services "my-rest-server" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=services, name=my-rest-server, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=services "my-rest-server" already exists, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={})."

Here is devfile:

---
apiVersion: 1.0.0
metadata:
  name: xpress
components:
  -
    type: cheEditor
    reference: https://raw.githubusercontent.com/cherxp/my-plugin-registry/master/meta.yaml
  -  
    type: chePlugin
    reference: https://raw.githubusercontent.com/cherxp/my-plugin-registry/master/meta-my-rest-server.yaml

[svkr2k]

Additional details (may not be helpful):

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
pvc-5bf1584a-2217-11ea-acba-029ec662a75d   1Gi        RWO            Delete           Bound    r1-che/claim-che-workspace   gp2                     13m
pvc-f63b8b0f-2216-11ea-acba-029ec662a75d   1Gi        RWO            Delete           Bound    che/postgres-data            gp2                     16m
pvc-f63b93f7-2216-11ea-acba-029ec662a75d   1Gi        RWO            Delete           Bound    che/keycloak-data            gp2                     16m
pvc-f63b9604-2216-11ea-acba-029ec662a75d   1Gi        RWO            Delete           Bound    che/keycloak-log             gp2                     16m

$ kubectl get pvc --all-namespaces
NAMESPACE   NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
che         keycloak-data         Bound    pvc-f63b93f7-2216-11ea-acba-029ec662a75d   1Gi        RWO            gp2            16m
che         keycloak-log          Bound    pvc-f63b9604-2216-11ea-acba-029ec662a75d   1Gi        RWO            gp2            16m
che         postgres-data         Bound    pvc-f63b8b0f-2216-11ea-acba-029ec662a75d   1Gi        RWO            gp2            16m
r1-che      claim-che-workspace   Bound    pvc-5bf1584a-2217-11ea-acba-029ec662a75d   1Gi        RWO            gp2            13m

[svkr2k]

@sparkoo , @tolusha , your opinions please ... thanks in advance.

[svkr2k]

Hi @tolusha , it would be very helpful if this issue can be added to sprint/current-sprint. I'm eagerly waiting for a solution for this. It would be nice if i can get some additional support on this. I apologize for interrupting.

[sparkoo]

@svkr2k I'm a bit lost here. We've identified an issue with chectl. Then you had issue with waiting for PVC and now issue with starting workspace (services "my-rest-server" already exists) ? Can we have this issue for chectl and create new issues for others?

Btw I'm getting 404 for the second plugin in your devfile in https://github.com/eclipse/che/issues/15440#issuecomment-566714454

[svkr2k]

Sorry, @sparkoo , thank you for the heads up ! i had the other questions while trying to figure out a solution for the my goal, that is, "create one workspace per user". I have removed unncessary comments (and shall move them to a separate issue if required).

Thank you for identifying the issue with chectl. As fixing the issue with chectl may take a while, I was hoping to try other workarounds that you had suggested and ran into other issues. Thank you very much for your support.

[sparkoo]

@svkr2k don't get me wrong. It's great that you've identified the problems and thank you for that. We just need to keep scope of one github issue clear, so it can be properly triaged and planned. If we find new issue during discussion, we should create new one so it can be again properly triaged.

To limit number of workspaces per user, there is che.limits.user.workspaces.count property https://github.com/eclipse/che/blob/master/assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/multiuser.properties#L56

[sparkoo]

fixed by https://github.com/che-incubator/chectl/pull/482