search

eclipse-archived / codewind

The official repository of the Eclipse Codewind project
https://codewind.dev
Eclipse Public License 2.0
113 stars 45 forks source link

SVT:Che:Creating projects getting into "Disabled" state and projects failed to build #2441

Closed jagraj closed 4 years ago

jagraj commented 4 years ago

Codewind version: 0.10.0 OS: CoreOS

Che version: 7.9.0 IDE extension version: 0.10.0 IDE version: Theia 7.9.0 Kubernetes cluster: OCP 4.3

Description: Codewind default WebSphere liberty project failed to build in Codewind 0.10.0 release.

image

pfe.txt

Steps to reproduce:

  1. Install Che 7.9 on OCP 4.3 cluster
  2. Create WebSphere liberty default project from Codewind 0.10 workspace.
  3. It fails to build the project.

Workaround:

jagraj commented 4 years ago

/priority stopship

jagraj commented 4 years ago

It builds fine on local but fails on Che.

malincoln commented 4 years ago

@elsony @rajivnathan pls assign. Thanks

jagraj commented 4 years ago

Looks like PFE is in bad state where all projects are failing now when I create new ones and they get into "Disabled" state. I did enable project but they are not getting enabled.

image

[09/03/20 23:04:44 springchecw010svt] [INFO] applog path:
[{"origin":"workspace","files":["/codewind-workspace/.logs/springchecw010svt-a4394d40-623c-11ea-b295-1d06f0782dd5/app.log"]}]
[09/03/20 23:04:44 /portal/modules/Project.js] [ERROR] { [Error: Unknown system error -122: Unknown system error -122, write]
  errno: -122,
  code: 'Unknown system error -122',
  syscall: 'write' }
log4js.fileAppender - Writing to file /codewind-workspace/.logs/Turbine.log, error happened  Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
    at doWrite (_stream_writable.js:411:19)
    at writeOrBuffer (_stream_writable.js:399:5)
    at WriteStream.Writable.write (_stream_writable.js:299:11)
    at RollingFileStream.BaseRollingFileStream._writeTheChunk (/file-watcher/server/node_modules/streamroller/lib/BaseRollingFileStream.js:50:25)
    at RollingFileStream.BaseRollingFileStream._write (/file-watcher/server/node_modules/streamroller/lib/BaseRollingFileStream.js:72:10)
    at doWrite (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:428:64)
    at writeOrBuffer (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:417:5)
    at RollingFileStream.Writable.write (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:334:11)
    at app (/file-watcher/server/node_modules/log4js/lib/appenders/file.js:55:12)
    at categoryAppenders.forEach (/file-watcher/server/node_modules/log4js/lib/log4js.js:42:5)
[09/03/20 23:04:54 wlchecw010svt] [INFO] Pod name was not found while looking up service information for project.

PFE log... pfe.txt

ssh24 commented 4 years ago

The issue seems to be here Disk quota exceeded:

[09/03/20 19:45:51 wlchecw010svt] [ERROR] executeBuildScript : 295 | tee: /codewind-workspace/.logs/wlchecw010svt-c0fb83d0-623c-11ea-b295-1d06f0782dd5/docker.build.log: Disk quota exceeded

log4js.fileAppender - Writing to file /codewind-workspace/.logs/Turbine.log, error happened  { [Error: Unknown system error -122: Unknown system error -122, write]
  errno: -122,
  code: 'Unknown system error -122',
  syscall: 'write' }
log4js.fileAppender - Writing to file /codewind-workspace/.logs/Turbine.log, error happened  Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
    at doWrite (_stream_writable.js:411:19)
    at writeOrBuffer (_stream_writable.js:399:5)
    at WriteStream.Writable.write (_stream_writable.js:299:11)
    at RollingFileStream.BaseRollingFileStream._writeTheChunk (/file-watcher/server/node_modules/streamroller/lib/BaseRollingFileStream.js:50:25)
    at RollingFileStream.BaseRollingFileStream._write (/file-watcher/server/node_modules/streamroller/lib/BaseRollingFileStream.js:72:10)
    at doWrite (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:428:64)
    at writeOrBuffer (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:417:5)
    at RollingFileStream.Writable.write (/file-watcher/server/node_modules/readable-stream/lib/_stream_writable.js:334:11)
    at app (/file-watcher/server/node_modules/log4js/lib/appenders/file.js:55:12)
    at categoryAppenders.forEach (/file-watcher/server/node_modules/log4js/lib/log4js.js:42:5)

Looks like che workspace ran out of memory limit?

johnmcollier commented 4 years ago

The Che workspace memory limit won't affect disk space. Additionally, PFE is technically running outside of the Che workspace and is not bound by any Che workspace quotas

It looks like the node that PFE is running on ran out of storage. @jagraj, can you provision a new OCP cluster and try there?

jagraj commented 4 years ago

Looks like persistent volume used 100% for codewind and it is causing this problem. I did create same number of projects in the past and did not hit 1GB limit. Is there option for the user to configure PV wth custom size.?

I am also wondering what changed in this release to consume entire volume.

172.30.146.217:6789:/volumes/csi/csi-vol-7dc2dbf0-6233-11ea-a88f-0a580afe100f  1.0G  1.0G     0 100% /var/lib/kubelet/pods/e0cbecff-e62d-4e27-80c4-ec33e083d21d/volume-subpaths/pvc-a8848592-45b6-46bf-8544-a2599ec5d5f1/codewind/0

NAME                                                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
persistentvolumeclaim/claim-che-workspace-workspacepg3r5u90kcx7o7yv   Bound    pvc-f6fe32ae-5029-474f-a35d-53a6158c4ccd   1Gi        RWO            rook-ceph-cephfs-internal   21h
persistentvolumeclaim/codewind-workspacepg3r5u90kcx7o7yv              Bound    pvc-a8848592-45b6-46bf-8544-a2599ec5d5f1   1Gi        RWX            rook-ceph-cephfs-internal   21h
persistentvolumeclaim/postgres-data                                   Bound    pvc-ef91a512-413a-4ee6-9f0a-860743c6356c   1Gi        RWO            rook-ceph-cephfs-internal   26h
johnmcollier commented 4 years ago

Hmm, there shouldn't have been anything in this release that increased volume usage.

@jagraj Which PVC is filled up? Is it codewind-workspacepg3r5u90kcx7o7yv or claim-che-workspace-workspacepg3r5u90kcx7o7yv, I'm guessing it's codewind-workspacepg3r5u90kcx7o7yv?

There's not much the user can do to configure the volume size currently (and this lack of configurability is why we need https://github.com/eclipse/codewind/issues/2447). We could increase the default volume size for Codewind on Che to 5Gi, rather than 1Gi.

johnmcollier commented 4 years ago

/assign

@elsony Any concerns about increasing our default volume size to 5Gi? FWIW, it's the same default odo uses 😉

jagraj commented 4 years ago

Hmm, there shouldn't have been anything in this release that increased volume usage.

@jagraj Which PVC is filled up? Is it codewind-workspacepg3r5u90kcx7o7yv or claim-che-workspace-workspacepg3r5u90kcx7o7yv, I'm guessing it's codewind-workspacepg3r5u90kcx7o7yv?

There's not much the user can do to configure the volume size currently (and this lack of configurability is why we need #2447). We could increase the default volume size for Codewind on Che to 5Gi, rather than 1Gi.

@johnmcollier Yes, it is codewind workspace volume and here is the list of projects by their sizes consuming in the volume.

216M    /codewind-workspace/wlchecw010svt1
2.0K    /codewind-workspace/.config
116K    /codewind-workspace/nodejschecw010svt
151M    /codewind-workspace/appsodynodejslbchecw010svt
36K     /codewind-workspace/swiftchecw010svt
4.5K    /codewind-workspace/gochecw010svt
1.1M    /codewind-workspace/cw-temp
26K     /codewind-workspace/appsodynodejsredchecw010svt
0       /codewind-workspace/appsodynodejschecw010svt
12M     /codewind-workspace/appsodynodejssimpchecw010svt
5.5K    /codewind-workspace/pythonchecw010svt
23M     /codewind-workspace/appsodynodejsscachecw010svt
216M    /codewind-workspace/wlchecw010svt
22K     /codewind-workspace/lagomchecw010svt
204M    /codewind-workspace/springchecw010svt
369K    /codewind-workspace/olchecw010svt
91K     /codewind-workspace/appsodyolchecw010svt
68M     /codewind-workspace/.extensions
4.2M    /codewind-workspace/.logs
0       /codewind-workspace/wlmpchecw010svt
512     /codewind-workspace/.appsody
0       /codewind-workspace/olcw10svt1
2.0K    /codewind-workspace/.projects
21K     /codewind-workspace/appsodyempchecw010svt
716K    /codewind-workspace/.idc
147M    /codewind-workspace/appsodykitchecw010svt
1.1G    /codewind-workspace/
elsony commented 4 years ago

/assign

@elsony Any concerns about increasing our default volume size to 5Gi? FWIW, it's the same default odo uses 😉

I think it is fine for us to change the default.

jagraj commented 4 years ago

Ok, I created Codewind 0.9 workspace with Che 7.9.0 version and I was able to create all projects one for each template and I did not hit space issue. The total space it is occupied is 553M (24 projects). I see WebSphere liberty project is taking 216MB vs 3.5M in 0.9.0. You can also see other projects as well where in Codewind 0.10.0 it is consuming more space almost double for just (18 projects).

553M    /codewind-workspace
204M    /codewind-workspace/springchecw09svt
151M    /codewind-workspace/appsodynodejslbchecw09svt
76M     /codewind-workspace/.extensions
75M     /codewind-workspace/appsodykitswiftchecw09svt
23M     /codewind-workspace/appsodynodejsscachecw09svt
12M     /codewind-workspace/appsodynodejssimpchecw09svt
5.2M    /codewind-workspace/.logs
3.6M    /codewind-workspace/wlchecw09svt
2.5M    /codewind-workspace/cw-temp
1018K   /codewind-workspace/odojavachecw09svt
710K    /codewind-workspace/.idc
369K    /codewind-workspace/olchecw09svt
219K    /codewind-workspace/odoperlchecw09svt
116K    /codewind-workspace/nodejschecw09svt
113K    /codewind-workspace/openshiftnodejschecw09svt
91K     /codewind-workspace/appsodyolchecw09svt
89K     /codewind-workspace/odopythonchecw09svt
81K     /codewind-workspace/.projects
36K     /codewind-workspace/swiftchecw09svt
26K     /codewind-workspace/appsodynodejsredchecw09svt
22K     /codewind-workspace/lagomchecw09svt
21K     /codewind-workspace/appsodyspringkotlinchecw09svt
21K     /codewind-workspace/appsodyempchecw09svt
17K     /codewind-workspace/appsodyspringchecw09svt
17K     /codewind-workspace/appsodynodejsdefchecw09svt
9.0K    /codewind-workspace/appsodypythonchecw09svt
5.5K    /codewind-workspace/pythonchecw09svt
5.0K    /codewind-workspace/appsodybashchecw09svt
4.5K    /codewind-workspace/gochecw09svt
2.0K    /codewind-workspace/.config
512     /codewind-workspace/.appsody
jagraj commented 4 years ago

Thanks @johnmcollier

I just verified this change and I do see default volume size is "5GB" now. Once we have doc changes and fix merged into master then I will go ahead and close this issue. We should consider this as an enhancement in future release to provide an option to the user to customize Codewind workspace size before creating one.

persistentvolumeclaim/codewind-workspaceqibvn8zhunt67rny              Bound    pvc-dbb259a9-7c9f-4835-bc26-93e4442229c2   5Gi        RWX            rook-ceph-cephfs-internal   2m12s
jagraj commented 4 years ago

/priority hot

jagraj commented 4 years ago

Making this as "hot" and this problem exist in previous releases and the new default size 5GB should help users with more projects with build cache.

malincoln commented 4 years ago

@jagraj have you verified the fix? It may have been closed automatically when 161 was closed.

jagraj commented 4 years ago

Yes, verified and it is closed.