search

kasmtech / workspaces-issues

18 stars 4 forks source link

Persistent profile error for new session #508

Open roma-ms opened 7 months ago

roma-ms commented 7 months ago

Existing Resources

Describe the bug New users cant create persistent profile when they do they try the get various error messages e.g gateway error

To Reproduce Steps to reproduce the behavior:

  1. Go to persistent profile "enabled"

  2. Click on ' launch session

  3. Scroll down to '....'

  4. See error 2024-01-22 11:52:45,256 [DEBUG] main.handler: Creating container image (sha256:1434dac6cef65194209b814e41fd2d1f57cec3381ee5ccaf6ad1203483489583) name (kasm_volume_helper_dd8485) to process volume mapping (/efs/kasm_user_share/) 2024-01-22 11:52:45,822 [DEBUG] main.handler: Processing of volume (/efs/kasm_user_share/) for container (a4bee85136740ff5ab9c5c49ef7f8ce27e76c046216fb8911e8a66abc2b9892d) succeeded. + RESET=False

    • mkdir -p /host_root/efs/*
    • '[' False == True ']'
    • chown 1000:1000 /host_root/efs/kasm_user_share/*
    • ls -ld /host_root/efs/kasm_user_share/ drwxrwxrwx. 16 kasm kasm 6144 Jan 17 12:56 /host_root/efs/kasm_user_share/

2024-01-22 11:52:45,824 [DEBUG] main.handler: Removing temporary volume container: (a4bee85136740ff5ab9c5c49ef7f8ce27e76c046216fb8911e8a66abc2b9892d) 2024-01-22 11:52:45,835 [DEBUG] main.handler: Persistent profile volume processing for (/efs/kasm_user_share/) succeeded 2024-01-22 11:52:46,259 [INFO] main.handler: container_networks dict_keys(['kasm_default_network']) 2024-01-22 11:52:46,263 [DEBUG] main.handler: kasm_proxy_networks dict_keys(['kasm_default_network']) 2024-01-22 11:52:46,264 [DEBUG] main.handler: Kasm proxy container already connected to network kasm_default_network 2024-01-22 11:52:46,268 [INFO] main.handler: Found kasm_proxy_ip (:443) for network (kasm_default_network) 2024-01-22 11:52:46,271 [DEBUG] main.handler: generate_nginx_config 2024-01-22 11:52:46,355 [DEBUG] main.handler: Nginx reload successful (2024/01/22 11:52:46 [warn] 711#711: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" 2024/01/22 11:52:46 [notice] 711#711: signal process started ) 2024-01-22 11:52:46,377 [DEBUG] main.handler: Executing kasm_post_run_user.sh 2024-01-22 11:52:46,464 [ERROR] main.handler: Error after container creation Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec: Conflict ("Container 44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc is restarting, wait until the container is running") Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec: Conflict ("Container 44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc is restarting, wait until the container is running") 2024-01-22 11:52:46,467 [DEBUG] main.handler: destroy 2024-01-22 11:52:46,816 [INFO] main.handler: Removing Existing Container 44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc 2024-01-22 11:52:46,817 [INFO] main.handler: Skipping shutdown script 2024-01-22 11:52:46,983 [DEBUG] main.handler: remove_nginx_config 2024-01-22 11:52:47,057 [DEBUG] main.handler: Nginx reload successful (2024/01/22 11:52:47 [warn] 733#733: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" 2024/01/22 11:52:47 [notice] 733#733: signal process started ) 2024-01-22 11:52:47,059 [DEBUG] main.handler: Searching for docker networks to prune 2024-01-22 11:52:47,059 [DEBUG] main.handler: get_container_by_id 2024-01-22 11:52:47,060 [DEBUG] main.handler: Getting Container (kasm_proxy) 2024-01-22 11:52:47,064 [INFO] main.handler: Container ID (44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc) Not Found 2024-01-22 11:52:47,066 [ERROR] main.handler: Exception creating Kasm: Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "init.py", line 481, in post File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec: Conflict ("Container 44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc is restarting, wait until the container is running") Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "init.py", line 481, in post File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc/exec: Conflict ("Container 44cc41e86c3b478f335abfa8a4ea881f8ac843421b5800230e87b67dc13c60fc is restarting, wait until the container is running")

2024-01-22 12:05:49,277 [DEBUG] main.handler: Image (xxxxxxxx) is present. No hash specified for verification 2024-01-22 12:05:49,278 [DEBUG] main.handler: provision 2024-01-22 12:05:49,279 [DEBUG] main.handler: Creating container image (sha256:1434dac6cef65194209b814e41fd2d1f57cec3381ee5ccaf6ad1203483489583) name (kasm_volume_helper_cebfbf) to process volume mapping (/efs/kasm_user_share/xxxxxxxxxx) 2024-01-22 12:05:49,831 [DEBUG] main.handler: Processing of volume (/efs/kasm_user_share/xxxxxxx) for container (3bf7506e30a871402d077abbc7478601d32f806cf3af6e162177cbcd2a2d80a3) succeeded. + RESET=False

2024-01-22 12:05:49,832 [DEBUG] main.handler: Removing temporary volume container: (3bf7506e30a871402d077abbc7478601d32f806cf3af6e162177cbcd2a2d80a3) 2024-01-22 12:05:49,843 [DEBUG] main.handler: Persistent profile volume processing for (/efs/kasm_user_share/****) succeeded 2024-01-22 12:05:50,256 [INFO] main.handler: container_networks dict_keys(['kasm_default_network']) 2024-01-22 12:05:50,260 [DEBUG] main.handler: kasm_proxy_networks dict_keys(['kasm_default_network']) 2024-01-22 12:05:50,260 [DEBUG] main.handler: Kasm proxy container already connected to network kasm_default_network 2024-01-22 12:05:50,265 [INFO] main.handler: Found kasm_proxy_ip (172.18.0.2:443) for network (kasm_default_network) 2024-01-22 12:05:50,266 [DEBUG] main.handler: generate_nginx_config 2024-01-22 12:05:50,355 [DEBUG] main.handler: Nginx reload successful (2024/01/22 12:05:50 [warn] 758#758: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" 2024/01/22 12:05:50 [notice] 758#758: signal process started ) 2024-01-22 12:05:50,385 [DEBUG] main.handler: Executing kasm_post_run_user.sh 2024-01-22 12:05:50,441 [ERROR] main.handler: Error after container creation Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec: Conflict ("Container 5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363 is restarting, wait until the container is running") Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec: Conflict ("Container 5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363 is restarting, wait until the container is running") 2024-01-22 12:05:50,446 [DEBUG] main.handler: destroy 2024-01-22 12:05:50,837 [INFO] main.handler: Removing Existing Container 5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363 2024-01-22 12:05:50,838 [INFO] main.handler: Skipping shutdown script 2024-01-22 12:05:51,007 [DEBUG] main.handler: remove_nginx_config 2024-01-22 12:05:51,087 [DEBUG] main.handler: Nginx reload successful (2024/01/22 12:05:51 [warn] 780#780: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/kasm_nginx.crt" 2024/01/22 12:05:51 [notice] 780#780: signal process started ) 2024-01-22 12:05:51,088 [DEBUG] main.handler: Searching for docker networks to prune 2024-01-22 12:05:51,089 [DEBUG] main.handler: get_container_by_id 2024-01-22 12:05:51,089 [DEBUG] main.handler: Getting Container (kasm_proxy) 2024-01-22 12:05:51,093 [INFO] main.handler: Container ID (5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363) Not Found 2024-01-22 12:05:51,094 [ERROR] main.handler: Exception creating Kasm: Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "init.py", line 481, in post File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec: Conflict ("Container 5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363 is restarting, wait until the container is running") Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 1021, in raise_for_status requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "init.py", line 481, in post File "provision.py", line 776, in provision File "docker/models/containers.py", line 193, in exec_run File "docker/utils/decorators.py", line 19, in wrapped File "docker/api/exec_api.py", line 80, in exec_create File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.44/containers/5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363/exec: Conflict ("Container 5f834a1f1d47c2952900c96d0dcd7a560395f4156962503e91d8efc92a824363 is restarting, wait until the container is running") Expected behavior A new session should open

Screenshots

Screenshot 2024-01-22 at 12 24 41

Workspaces Version e.g Version 1.12

Workspaces Installation Method terraform

Client Browser (please complete the following information):

Workspace Server Information (please provide the output of the following commands):

Additional context Add any other context about the problem here.

j-travis commented 7 months ago

I recommend reading over our NFS setup guide, in particular the specifics we demonstrate in the /etc/exports https://kasmweb.com/docs/latest/guide/persistent_data/host_guides.html#host-configuration-guides

And also the tests for creating a file via sudo and verify the permissions are correct on the file after being created (1000)

You'll want to verify you can setup EFS or the mount with the all_squash,anonuid=1000,anongid=1000,no_subtree_check or equivalent

Once you get that working, try creating a new session.

If it still fails you can use these steps to take a look at the session container logs: https://github.com/kasmtech/workspaces-issues/issues/444#issuecomment-1736460285