SVT:[Windows]:[Hybrid]: Appsody Quarkus sample failed to deploy on remote Codewind

[jagraj]

Codewind version: Codewind 0.11 OS: Windows/OCP 4.3

Che version: IDE extension version: Codewind 0.11.0 IDE version: Eclipse 2019-09 Kubernetes cluster: OpenShift 4.3

Description: I installed codewind on remote OCP 4.3 cluster and configured Codewind remote connection on Eclipse and tried to create Appsody Quarkus project and it failed to build with the following exception.

Running development environment...
Pulling docker image docker.io/appsody/quarkus:0.3
Running command: buildah pull docker.io/appsody/quarkus:0.3
03ee3603d3b5ebdedf4559e7f02eccd345b29b5dc052b61096d4914b37597efc
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/appsodyquahybcw011svt/app-deploy.yaml
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/appsodyquahybcw011svt/app-service.yaml
Getting the logs ...
Running command: kubectl logs deployment/cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7" in pod "cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7-74d485c8526gfx" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7" in pod "cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7-74d485c8526gfx" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7 -f "--pod-running-timeout=2m"
[Container] Running command:  ./mvnw -B compile quarkus:dev
[Container] /bin/sh: ./mvnw: Permission denied
[Container] [Error] Wait received error with status code: 126 due to error: exit status 126

cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7-74d485c8526gfx   0/1     CrashLoopBackOff   11         34m

oc logs cw-appsodyquahybcw011svt-704e0b90-80cf-11ea-a7a7-74d485c8526gfx
Running command:  ./mvnw -B compile quarkus:dev
/bin/sh: ./mvnw: Permission denied
[Error] Wait received error with status code: 126 due to error: exit status 126

Logs gate-keeper.txt performance.txt pfe.txt .log

Steps to reproduce:

1. Install remote codewind on OCP 4.3
2. Create a connection from Eclipse plugin.
3. Create Appsody Quarkus Project
4. It fails to build with the above failure.

Workaround:

[jagraj]

/priority hot

[naveenkaratekid]

Codewind version: 0.11.0 OS: Windows w. OKD/OpenShift cluster IDE: Eclipse 2019-06 (4.12.0) IDE extension version: 0.11.0

I am noticing this issue in my Windows VM for Eclipse remote

Logs from Eclipse in Windows:

Pulling docker image docker.io/appsody/quarkus:0.3
Running command: buildah pull docker.io/appsody/quarkus:0.3
03ee3603d3b5ebdedf4559e7f02eccd345b29b5dc052b61096d4914b37597efc
Successfully added your project to /codewind-workspace/.appsody/project.yaml
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/remote-apps-quarkus/app-deploy.yaml
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/remote-apps-quarkus/app-service.yaml
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-remoteappsquarkus-d705b900-80d4-11ea-8353" in pod "cw-remoteappsquarkus-d705b900-80d4-11ea-8353-d668ff78f-4r8vk" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-remoteappsquarkus-d705b900-80d4-11ea-8353" in pod "cw-remoteappsquarkus-d705b900-80d4-11ea-8353-d668ff78f-4r8vk" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-remoteappsquarkus-d705b900-80d4-11ea-8353" in pod "cw-remoteappsquarkus-d705b900-80d4-11ea-8353-d668ff78f-4r8vk" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Error from server (BadRequest): container "cw-remoteappsquarkus-d705b900-80d4-11ea-8353" in pod "cw-remoteappsquarkus-d705b900-80d4-11ea-8353-d668ff78f-4r8vk" is waiting to start: PodInitializing
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Running command:  ./mvnw -B compile quarkus:dev
[Container] /bin/sh: ./mvnw: Permission denied
[Container] [Error] Wait received error with status code: 126 due to error: exit status 126
Running development environment...
Pulling docker image docker.io/appsody/quarkus:0.3
Running command: buildah pull docker.io/appsody/quarkus:0.3
03ee3603d3b5ebdedf4559e7f02eccd345b29b5dc052b61096d4914b37597efc
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/remote-apps-quarkus/app-deploy.yaml
Attempting to apply resource in Kubernetes ...
Running command: kubectl apply -f /codewind-workspace/remote-apps-quarkus/app-service.yaml
Getting the logs ...
Running command: kubectl logs deployment/cw-remoteappsquarkus-d705b900-80d4-11ea-8353 -f "--pod-running-timeout=2m"
[Container] Running command:  ./mvnw -B compile quarkus:dev
[Container] /bin/sh: ./mvnw: Permission denied
[Container] [Error] Wait received error with status code: 126 due to error: exit status 126

Screenshot:

Strangely, I don't see that problem in my remote connection in Eclipse in Mac OS

[sujeilyfonseca]

Codewind version: 0.11.0 OS: Windows w. OKD/OpenShift cluster

IDE extension version: 0.11.0 IDE version: Code 1.44.0

I was able to reproduce a similar scenario using hybrid VS Code:

cw-cwremoteappsodyquarkus-c5bae250-80d5-11ea-ad1e-5499b8696z78w   0/1       Error     8          17m

PS C:\Users\Administrator\.codewind\0.11.0> oc logs cw-cwremoteappsodyquarkus-c5bae250-80d5-11ea-ad1e-5499b8696z78w                                                                                                                                                                 Running command:  ./mvnw -B compile quarkus:dev
/bin/sh: ./mvnw: Permission denied
[Error] Wait received error with status code: 126 due to error: exit status 126

remote-codewind-gatekeeper.txt remote-codewind-pfe.txt

[DavidG1011]

Codewind version: 0.11.0 OS: MacOS Cluster: OKD 4.2

IDE extension version: 0.11.0 IDE version: Code 1.44.0

Not able to reproduce this on MacOS connected to OKD 4.2

Logs in case they're useful: remote-gatekeeper.txt remote-pfe.txt

[sujeilyfonseca]

Codewind version: 0.11.0 OS: Mac OS Mojave 10.14.6 w. OKD/OpenShift cluster

IDE extension version: 0.11.0 IDE version: Code 1.44.0

I was not able to reproduce the problem in Mac OS, using VS Code connected to a Fyre Cluster with OCP 4.3:

[makandre]

This is possibly related to or a symptom of https://github.com/eclipse/codewind/issues/875; maybe we didn't consider when the starting point is a project on Windows.

[malincoln]

@tobespc pls investigate. We can discuss this next week.

[elsony]

@tobespc Would you mind to provide more info on what part do you want my team to investigate on? It's the portal code that does the sync and the mount point, is there anything specific that you find is related to iterative dev on this issue?

[tobespc]

@elsony we've not dug too deeply into it yet but as this also mentioned deploying and problems with logs it might be something on your side as well

[elsony]

@tobespc The build problem is rooted from the permission problem which usually indicates either there is a permission issue when using a specific stack or there are permission problems when syncing the resources to container. Therefore, I suggest the portal team/appsody team to resolve the permission problem before passing on it iterative dev for investigation.

[jagraj]

Yes, some how mvnw file does not have execute permissions after project source is extracted. After giving execute permissions then the quarkus project built fine.

[mattcolegate]

/assign /pipeline In Progress

[mattcolegate]

Recreated the issue and I can see that execute permissions are set on mvnw in the project source locally, but in codewind-workspace on PFE it's 666 - no execute permissions. I'm starting my investigations in cwctl to check that file permissions are being sent to PFE correctly.

[mattcolegate]

The output from cwctl was as follows:

May 7 11:43:43.511 AM [13596.954] Successfully ran installer command: [C:\Users\MattColegate\.codewind\0.12.0\cwctl.exe] [--insecure] [project] [sync] [-p] [c:\Users\MattColegate\Desktop\cw-eclipseDev-workspace\mattAppQuaDef2] [-i] [962c1930-904f-11ea-aa68-1b95e7474753] [-t] [1588848208014] 
May 7 11:43:43.512 AM [13596.954] Std(out/err):{false 438 .appsody-config.yaml }
{false 438 .cw-settings }
{false 438 mvnw }
{false 438 mvnw.cmd }
{false 438 pom.xml }
{false 438 src/main/java/org/acme/quickstart/GreetingResource.java }
{false 438 src/main/java/org/acme/quickstart/GreetingService.java }
{false 438 src/main/resources/META-INF/resources/index.html }
{false 438 src/test/java/org/acme/quickstart/GreetingResourceTest.java }
{false 438 src/test/java/org/acme/quickstart/NativeGreetingResourceIT.java }
Status: 200 OK

438 is the decimal representation of 0x666, so I think it's a problem with the getting of the file permissions on Windows with golang. Will attempt to fix.

[mattcolegate]

I've finished problem determination. File permissions are not accurate when using Go on Windows due to Windows using Access Control Lists to regulate file access rather than unix-style permission bits. There is no quick solution for this - solutions discussed include:

• defaulting all files uploaded from Windows to all-permissions-on
• maintaining a white list of files for PFE to change permissions after upload
• zipping source files before transmission, which would hopefully preserve permissions
• postponing appsody init until after transmission

/unassign /pipeline portal backlog

[mattcolegate]

@tobespc do you think it's worth discussing this one at tech topics tomorrow? There doesn't seem to be a clear way forward.

[rwalle61]

/pipeline Verify