search

eclipse-archived / codewind

The official repository of the Eclipse Codewind project
https://codewind.dev
Eclipse Public License 2.0
113 stars 45 forks source link

Kube Project Delete Fails due to Log Dir #1304

Open maysunfaisal opened 4 years ago

maysunfaisal commented 4 years ago

Codewind version: OS:

Che version: IDE extension version: IDE version: Kubernetes cluster:

Description: On Functional Tests, I see

[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] operationId: 2ed0936e78e860d1b6318981b09db419
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] containerDelete: Kill running processes and remove container... 
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] Project ID:        default-nodejs-1574910173921
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] Project Location:  /codewind-workspace/turbinetest-default-nodejs
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] Project Type:      nodejs
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] Project Container: cw-turbinetestdefaultnodej-default-nodejs-15749101
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Retrieved project information for project /file-watcher/fwdata/projects/default-nodejs-1574910173921/default-nodejs-1574910173921.json
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] {"projectID":"default-nodejs-1574910173921","projectType":"nodejs","location":"/codewind-workspace/turbinetest-default-nodejs","autoBuildEnabled":true,"startMode":"run","appPorts":["3000"],"language":"nodejs","debugPort":"9229","ignoredPaths":["*/node_modules*","*/.git/*","*/.DS_Store","*/.dockerignore","*/.gitignore"],"isHttps":false,"appBaseURL":"http://cw-turbinetestdefaultnodej-default--che-che.9.46.125.238.nip.io","sentProjectInfo":true,"forceAction":"REBUILD","contextRoot":"/","healthCheck":"/"}
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Project location: /codewind-workspace/turbinetest-default-nodejs
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Retrieved project information for project /file-watcher/fwdata/projects/default-nodejs-1574910173921/default-nodejs-1574910173921.json
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] {"projectID":"default-nodejs-1574910173921","projectType":"nodejs","location":"/codewind-workspace/turbinetest-default-nodejs","autoBuildEnabled":true,"startMode":"run","appPorts":["3000"],"language":"nodejs","debugPort":"9229","ignoredPaths":["*/node_modules*","*/.git/*","*/.DS_Store","*/.dockerignore","*/.gitignore"],"isHttps":false,"appBaseURL":"http://cw-turbinetestdefaultnodej-default--che-che.9.46.125.238.nip.io","sentProjectInfo":true,"forceAction":"REBUILD","contextRoot":"/","healthCheck":"/"}
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Project location: /codewind-workspace/turbinetest-default-nodejs
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Deleting project default-nodejs-1574910173921
[28/11/19 03:26:20 turbinetest-default-nodejs] [INFO] projectInfo.deploymentRegistry: undefined
[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] *** NODE.JS

[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] *** PWD = /file-watcher/server
*** ROOT = /codewind-workspace/turbinetest-default-nodejs
*** LOCAL_WORKSPACE = /codewind-workspace
*** PROJECT_ID = default-nodejs-1574910173921
*** COMMAND = remove

[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] *** CONTAINER_NAME = cw-turbinetestdefaultnodej-default-nodejs-15749101
*** AUTO_BUILD_ENABLED = undefined
*** LOGNAME = undefined
*** START_MODE = undefined

[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] *** DEBUG_PORT = undefined
*** FORCE_ACTION = undefined
*** LOG_FOLDER = /codewind-workspace/.logs/undefined
*** DEPLOYMENT_REGISTRY = undefined
*** HOST_OS = 

[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] project=cw-turbinetestdefaultnodej-default-nodejs-15749101

[28/11/19 03:26:20 turbinetest-default-nodejs] [TRACE] Removing the container for app /codewind-workspace/turbinetest-default-nodejs.

[28/11/19 03:26:21 Turbine] [TRACE] [turbinetest-default-nodejs] Running processes:
[28/11/19 03:26:21 Turbine] [TRACE] [turbinetest-default-nodejs]    PID   PPID   PGID COMMAND
     1      0      1 sh
    97      1      1 npm
   113     97      1 npm-start.sh
   114    113      1 npm
   138    114      1 node
  3870      0   3870 bash
  3881   3870   3870 npm
  3917   3881   3870 node
  4087   3917   3870 node
  4136   4087   3870 node
 48462      0  48117 kubectl
 48481   4136  48481 nodejs-containe
 48483  48481  48481 helm
 48492   4136   3870 ps

[28/11/19 03:26:24 turbinetest-default-nodejs] [TRACE] release "cw-turbinetestdefaultnodej-default-nodejs-15749101" deleted

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] untagged: localhost/cw-turbinetestdefaultnodej-default-nodejs-15749101:latest

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] 9283ab449b1b19e79b40e7150a5da4a931c08fbdfea35761f2369c41f80de687

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] b9721b35708045f4143bbb5620b73c9901f89900aec8d098f808af0b727cff5c

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] 3b89812bdf740147517f025d6e025c584234f13290340aabd34ed54e6ce45743

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] 1717487c3ba8acb932c0c38eef193ee09101d1e608ea63dc929081d3473241da

[28/11/19 03:26:25 turbinetest-default-nodejs] [TRACE] 601724a1e5836ad997dbeff081b612819ef9b8c924186ed599259e4dbf91c723

[28/11/19 03:26:25 turbinetest-default-nodejs] [INFO] Removing dangling images for default-nodejs-1574910173921
[28/11/19 03:26:25 turbinetest-default-nodejs] [INFO] Project info cache is {}
[28/11/19 03:26:25 turbinetest-default-nodejs] [INFO] Successfully deleted default-nodejs-1574910173921
[28/11/19 03:26:25 Turbine] [INFO] Removing project logs directory
[28/11/19 03:26:25 Turbine] [ERROR] [File Name: /file-watcher/server/src/utils/logger.ts | Function Name: logProjectError | Line Number: 56] default-nodejs-1574910173921: Failed to remove project default-nodejs-1574910173921
[28/11/19 03:26:25 Turbine] [ERROR] [File Name: /file-watcher/server/src/utils/logger.ts | Function Name: logProjectError | Line Number: 56] default-nodejs-1574910173921: Error: ENOTEMPTY: directory not empty, rmdir '/codewind-workspace/.logs/turbinetest-default-nodejs-default-nodejs-1574910173921'
[28/11/19 03:26:25 Turbine] [INFO] default-nodejs-1574910173921: Emitting event 
 message: projectDeletion
 data: {
  "operationId": "2ed0936e78e860d1b6318981b09db419",
  "projectID": "default-nodejs-1574910173921",
  "status": "failed",
  "error": "Failed to remove project default-nodejs-1574910173921"
}
      1) delete project
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] operationId: 28aae8e715ac160aae2fb1dd1fb68895
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] containerDelete: Kill running processes and remove container... 
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] Project ID:        default-liberty-1574910173921
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] Project Location:  /codewind-workspace/turbinetest-default-liberty
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] Project Type:      liberty
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] Project Container: cw-turbinetestdefaultliber-default-liberty-1574910
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Retrieved project information for project /file-watcher/fwdata/projects/default-liberty-1574910173921/default-liberty-1574910173921.json
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] {"projectID":"default-liberty-1574910173921","projectType":"liberty","location":"/codewind-workspace/turbinetest-default-liberty","autoBuildEnabled":true,"startMode":"run","appPorts":["9080"],"language":"liberty","debugPort":"7777","ignoredPaths":["*/node_modules*","*/.git/*","*/.DS_Store","*/.dockerignore","*/.gitignore"],"isHttps":false,"appBaseURL":"http://cw-turbinetestdefaultliber-default--che-che.9.46.125.238.nip.io","sentProjectInfo":true,"forceAction":"REBUILD","contextRoot":"/","healthCheck":"/","mavenProfiles":[],"mavenProperties":[]}
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Project location: /codewind-workspace/turbinetest-default-liberty
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Retrieved project information for project /file-watcher/fwdata/projects/default-liberty-1574910173921/default-liberty-1574910173921.json
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] {"projectID":"default-liberty-1574910173921","projectType":"liberty","location":"/codewind-workspace/turbinetest-default-liberty","autoBuildEnabled":true,"startMode":"run","appPorts":["9080"],"language":"liberty","debugPort":"7777","ignoredPaths":["*/node_modules*","*/.git/*","*/.DS_Store","*/.dockerignore","*/.gitignore"],"isHttps":false,"appBaseURL":"http://cw-turbinetestdefaultliber-default--che-che.9.46.125.238.nip.io","sentProjectInfo":true,"forceAction":"REBUILD","contextRoot":"/","healthCheck":"/","mavenProfiles":[],"mavenProperties":[]}
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Project location: /codewind-workspace/turbinetest-default-liberty
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Deleting project default-liberty-1574910173921
[28/11/19 03:16:44 turbinetest-default-liberty] [INFO] projectInfo.deploymentRegistry: undefined
[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] *** JAVA
*** PWD = /file-watcher/server
*** ROOT = /codewind-workspace/turbinetest-default-liberty
*** LOCAL_WORKSPACE = /codewind-workspace
*** PROJECT_ID = default-liberty-1574910173921
*** COMMAND = remove
*** CONTAINER_NAME = cw-turbinetestdefaultliber-default-liberty-1574910
*** START_MODE = undefined
*** DEBUG_PORT = undefined
*** HOST_OS = 
*** DEPLOYMENT_REGISTRY = undefined
*** MAVEN_SETTINGS = 

[28/11/19 03:16:44 turbinetest-default-liberty] [TRACE] Removing the container for app /codewind-workspace/turbinetest-default-liberty.

[28/11/19 03:16:45 turbinetest-default-liberty] [TRACE] Configuration file detected: 
[28/11/19 03:16:45 turbinetest-default-liberty] [TRACE] 

[28/11/19 03:16:45 turbinetest-default-liberty] [TRACE] libertyEnv : microclimate
appPrefix : microclimate-dev-liberty-
idcBasePath : /codewind-workspace

[28/11/19 03:16:45 turbinetest-default-liberty] [TRACE] * Stopping and removing application: cw-turbinetestdefaultliber-default-liberty-1574910

[28/11/19 03:16:45 Turbine] [TRACE] [turbinetest-default-liberty] Running processes:
[28/11/19 03:16:45 Turbine] [TRACE] [turbinetest-default-liberty]    PID   PPID   PGID COMMAND
     1      0      1 sh
    97      1      1 npm
   113     97      1 npm-start.sh
   114    113      1 npm
   138    114      1 node
  3870      0   3870 bash
  3881   3870   3870 npm
  3917   3881   3870 node
  4087   3917   3870 node
  4136   4087   3870 node
 33928      0  11832 idc
 33934  33928  11832 java
 33939      0  11832 kubectl
 33977  33934  11832 kubectl
 36232   4136  36232 liberty-contain
 36235  36232  36232 idc
 36239  36235  36232 java
 36261  36239  36232 helm
 36273   4136   3870 ps

[28/11/19 03:16:46 turbinetest-default-liberty] [TRACE] Successfully removed the Helm release cw-turbinetestdefaultliber-default-liberty-1574910

[28/11/19 03:16:46 turbinetest-default-liberty] [TRACE] Awaiting resource deleting confirmation for release cw-turbinetestdefaultliber-default-liberty-1574910

[28/11/19 03:17:18 turbinetest-default-liberty] [TRACE] K8 pod for release cw-turbinetestdefaultliber-default-liberty-1574910has been terminated

[28/11/19 03:17:19 turbinetest-default-liberty] [TRACE] Successfully removed the application image cw-turbinetestdefaultliber-default-liberty-1574910
* Successfully removed the application

[28/11/19 03:17:19 turbinetest-default-liberty] [TRACE] Deleting /codewind-workspace/.idc/db/db-7a9469c69dd60b3a3158f1930829cfb924ace300

[28/11/19 03:17:19 turbinetest-default-liberty] [TRACE] The corresponding IDC container metadata content was deleted.
Deleting /codewind-workspace/.idc/docker/docker-7a9469c69dd60b3a3158f1930829cfb924ace300

[28/11/19 03:17:19 turbinetest-default-liberty] [TRACE] The corresponding IDC image metadata content was deleted.

[28/11/19 03:17:19 turbinetest-default-liberty] [TRACE] /file-watcher/server

[28/11/19 03:17:19 turbinetest-default-liberty] [INFO] Removing dangling images for default-liberty-1574910173921
[28/11/19 03:17:19 turbinetest-default-liberty] [INFO] Project info cache is {}
[28/11/19 03:17:19 turbinetest-default-liberty] [INFO] Successfully deleted default-liberty-1574910173921
[28/11/19 03:17:19 Turbine] [INFO] Removing project logs directory
[28/11/19 03:17:19 Turbine] [ERROR] [File Name: /file-watcher/server/src/utils/logger.ts | Function Name: logProjectError | Line Number: 56] default-liberty-1574910173921: Failed to remove project default-liberty-1574910173921
[28/11/19 03:17:19 Turbine] [ERROR] [File Name: /file-watcher/server/src/utils/logger.ts | Function Name: logProjectError | Line Number: 56] default-liberty-1574910173921: Error: ENOTEMPTY: directory not empty, rmdir '/codewind-workspace/.logs/turbinetest-default-liberty-default-liberty-1574910173921'
[28/11/19 03:17:19 Turbine] [INFO] default-liberty-1574910173921: Emitting event 
 message: projectDeletion
 data: {
  "operationId": "28aae8e715ac160aae2fb1dd1fb68895",
  "projectID": "default-liberty-1574910173921",
  "status": "failed",
  "error": "Failed to remove project default-liberty-1574910173921"
}
      1) delete project

I think there may be a 0.6 issue for the same thing, I will try searching and link it to this

maysunfaisal commented 4 years ago

Maybe related: https://github.com/eclipse/codewind/issues/640

Looks the same error stack

makandre commented 4 years ago

I'm seeing this in hybrid also.