Git cloning fails behind a proxy

yeungalan0 commented 3 years ago

Describe the bug

When setting up a new eclipse che application using the standard operator with all default settings on an OpenShift cluster behind a proxy with a self signed certificate, everything seems to work except that after workspace creation the initial git clone times out, which seems to indicate it's not honoring/aware of the proxies.

On the theia container the proxy Env variables seem to be set
Manually I can run a git clone on the theia container in my project directory and that succeeds without issue
Image pulling and everything else seems to succeed in eclipse che, indicating that the cluster proxies are respected, but just the plugin/process that does the initial git clone seems to ignore them

Che version

[x] latest
[ ] nightly
[ ] other: please specify

Eclipse-che operator version: 7.27.0

Steps to reproduce

Deploy Eclipse Che using the Operator (leaving all defaults) to an OpenShift cluster with proxy configuration
Start a workspace
Project cloning fails with a timeout

Expected behavior

Cloning should succeed and pickup the proxy variables set in the environment.

Runtime

[ ] kubernetes (include output of kubectl version)
[x] Openshift (include output of oc version)
[ ] minikube (include output of minikube version and kubectl version)
[ ] minishift (include output of minishift version and oc version)
[ ] docker-desktop + K8S (include output of docker version and kubectl version)
[ ] other: (please specify)

Screenshots

Installation method

[ ] chectl
- provide a full command that was used to deploy Eclipse Che (including the output)
- provide an output of chectl version command
[x] OperatorHub
[ ] I don't know

Environment

[ ] my computer
- [ ] Windows
- [ ] Linux
- [ ] macOS
[ ] Cloud
- [ ] Amazon
- [ ] Azure
- [ ] GCE
- [ ] other (please specify)
[x] other: OpenShift cluster

Eclipse Che Logs

Log output from theia container:

2021-03-09 01:08:13.066 root ERROR [hosted-plugin: 46] Child process git stderr: fatal: unable to access 'https://github.com/che-samples/python-hello-world.git/': Failed to connect to github.com port 443: Operation timed out 
2021-03-09 01:08:13.066 root ERROR Child process git stderr: fatal: unable to access 'https://github.com/che-samples/python-hello-world.git/': Failed to connect to github.com port 443: Operation timed out

2021-03-09 01:08:13.189 root ERROR [hosted-plugin: 46] Child process "git" exited with code 128 
2021-03-09 01:08:13.189 root ERROR Child process "git" exited with code 128

2021-03-09 01:08:13.191 root INFO [hosted-plugin: 46] Couldn't clone https://github.com/che-samples/python-hello-world.git Error: Cloning into '/projects/python-hello-world'...
fatal: unable to access 'https://github.com/che-samples/python-hello-world.git/': Failed to connect to github.com port 443: Operation timed out

    at ChildProcess.<anonymous> (/tmp/theia-unpacked/eclipse_che_workspace_plugin.theia/lib/exec.js:83:36)
    at ChildProcess.emit (events.js:314:20)
    at maybeClose (internal/child_process.js:1022:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5) 
2021-03-09 01:08:13.191 root INFO Couldn't clone https://github.com/che-samples/python-hello-world.git Error: Cloning into '/projects/python-hello-world'...
fatal: unable to access 'https://github.com/che-samples/python-hello-world.git/': Failed to connect to github.com port 443: Operation timed out

    at ChildProcess.<anonymous> (/tmp/theia-unpacked/eclipse_che_workspace_plugin.theia/lib/exec.js:83:36)
    at ChildProcess.emit (events.js:314:20)
    at maybeClose (internal/child_process.js:1022:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

2021-03-09 01:08:14.192 root ERROR [hosted-plugin: 46] Promise rejection not handled in one second: Error: Error: Cloning into '/projects/python-hello-world'...

We also see the below in the theia container logs, but aren't sure if it's related...

2021-03-09 01:06:03.383 root ERROR [hosted-plugin: 46] Promise rejection not handled in one second: TypeError: Cannot read property 'path' of undefined , reason: TypeError: Cannot read property 'path' of undefined 
2021-03-09 01:06:03.383 root ERROR Promise rejection not handled in one second: TypeError: Cannot read property 'path' of undefined , reason: TypeError: Cannot read property 'path' of undefined

2021-03-09 01:06:03.384 root ERROR [hosted-plugin: 46] With stack trace: TypeError: Cannot read property 'path' of undefined
    at Object.a [as joinPath] (/default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:248754)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116514
    at Array.forEach (<anonymous>)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116371
    at Array.forEach (<anonymous>)
    at f (/default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116229)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:122454 
2021-03-09 01:06:03.384 root ERROR With stack trace: TypeError: Cannot read property 'path' of undefined
    at Object.a [as joinPath] (/default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:248754)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116514
    at Array.forEach (<anonymous>)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116371
    at Array.forEach (<anonymous>)
    at f (/default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:116229)
    at /default-theia-plugins/vscode-builtin-json-language-features/extension/client/dist/node/jsonClientMain.js:1:122454

Additional context

Related issue: #17017 - This closed issue seems to match what we're seeing exactly

Release Notes

Cloning a git repository failed even if the proxy was configured at the container level. This has been resolved when the editor is Che-Theia.

tolusha commented 3 years ago

I've just tried to deploy Eclipse Che 7.27 on OpenShift 4.7 with cluster wide proxy configured. I've set nonProxyHosts due to https://github.com/eclipse/che/issues/17681

nonProxyHosts: 'api.<DOMAIN>|oauth-openshift.apps.<DOMAIN>'

I was able to start a workspace and project was successfully cloned.

It leads me to question if it is possible to check proxy server logs?

@azatsarynnyy I am wondering if curl is used to clone the project. If so, @yeungalan0 could you check if curl works correctly inside theia container?

yeungalan0 commented 3 years ago

Thanks for looking into this @tolusha !

Ok we're using Eclipse Che 7.27 on Openshift 4.5.31 with a cluster wide proxy configured.

We used the default operator settings (so we didn't set the nonProxyHosts manually).

Unfortunately, we do not have access to the proxy server logs to see if the request even hit it.

curl seems to work without issue on the theia container.

bash-5.0$ curl www.google.com
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/logos/doodles/2021/wu-lien-tehs-142nd-birthday-6753651837108881-l.png" itemprop="image"><meta content="Dr. Wu Lien-teh's 142nd Birthday" 
...

The default operator settings we used to standup the cluster:

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  auth:
    identityProviderURL: ''
    identityProviderRealm: ''
    oAuthSecret: ''
    identityProviderPassword: ''
    identityProviderImage: ''
    oAuthClientName: ''
    initialOpenShiftOAuthUser: true
    identityProviderClientId: ''
    identityProviderAdminUserName: ''
    externalIdentityProvider: false
  database:
    postgresImage: ''
    chePostgresUser: ''
    externalDb: false
    chePostgresHostName: ''
    chePostgresPassword: ''
    chePostgresDb: ''
    chePostgresPort: ''
  metrics:
    enable: true
  server:
    proxyURL: ''
    cheClusterRoles: ''
    singleHostGatewayConfigMapLabels: {}
    singleHostGatewayImage: ''
    proxyPassword: ''
    nonProxyHosts: ''
    pluginRegistryImage: ''
    serverMemoryRequest: ''
    devfileRegistryImage: ''
    proxyPort: ''
    singleHostGatewayConfigSidecarImage: ''
    tlsSupport: true
    serverMemoryLimit: ''
    allowUserDefinedWorkspaceNamespaces: false
    serverTrustStoreConfigMapName: ''
    proxyUser: ''
    cheImage: ''
    cheWorkspaceClusterRole: ''
    workspaceNamespaceDefault: <username>-che
    serverExposureStrategy: ''
    gitSelfSignedCert: false
    useInternalClusterSVCNames: true
    cheFlavor: ''
    cheImageTag: ''
  storage:
    postgresPVCStorageClassName: ''
    preCreateSubPaths: true
    pvcClaimSize: 1Gi
    pvcJobsImage: ''
    pvcStrategy: common
    workspacePVCStorageClassName: ''

yeungalan0 commented 3 years ago

So seems like the OpenShift cluster wide proxy settings are automatically set in the theia container (including the no_proxy). When I unset the proxies and then run a clone, the timeout issue appears. Which seems to indicate that the module that does the initial git clone just seems to not have inherited the proxy Env variables somehow...

bash-5.0$ unset https_proxy http_proxy
bash-5.0$ git clone https://github.com/che-samples/python-hello-world.git
Cloning into 'python-hello-world'...
fatal: unable to access 'https://github.com/che-samples/python-hello-world.git/': Failed to connect to github.com port 443: Operation timed out

tolusha commented 3 years ago

Your suggestions make sense... I don't see that env variables are used to clone the project [1] [2] [3] @azatsarynnyy it sounds like a bug

[1] https://github.com/eclipse/che-theia/blob/master/plugins/workspace-plugin/src/git.ts#L101 [2] https://github.com/eclipse/che-theia/blob/master/plugins/workspace-plugin/src/exec.ts#L15 [3] https://github.com/eclipse/che-theia/blob/master/plugins/workspace-plugin/src/exec.ts#L55

RomanNikitenko commented 3 years ago

Taking into account https://github.com/eclipse/che/issues/19242#issuecomment-796641839 looks like the problem is on workspace-plugin side, so I'm changing the area to plugins.

tolusha commented 3 years ago

/cc @svor

azatsarynnyy commented 3 years ago

@azatsarynnyy it sounds like a bug

@tolusha I agree

sunix commented 3 years ago

che-theia is invoking the git command. So if the env variable is there it should work ... we'll investigate

vitaliy-guliy commented 3 years ago

@yeungalan0 could you create a workspace from the devfile on your che deployment in a proxy environment and test how cloning works?

There is a PR with workspace plugin improvements. Now the plugin checks for http_proxy, https_proxy and no_proxy environment variables and applies them when it runs git or ssh.

vitaliy-guliy commented 3 years ago

See the PR to get more details how the fixup was tested.

eclipse-che / che