cloudfoundry / app-autoscaler-release

Automated scaling for apps running on Cloud Foundry
Apache License 2.0
26 stars 50 forks source link

create-release scripts not honoring proxy? #111

Closed cwb124 closed 4 years ago

cwb124 commented 6 years ago

I have a linux box with no internet access but has access via a proxy. I have set http_proxy and https_proxy on the box and I can curl -x successfully through the proxy. However when I run bosh create-release it gets a good chunk of the way through but then dies on the apiserver part: Added package 'apiserver/b0ab0e6e8317cd7292c8230d491e13f232f623c7'

It looks like those scripts are not honoring the proxy? I was watching my squid proxy logs throughout the create-release process and it was getting a lot of traffic until the apiserver portion started, and then there wasn't another entry in the proxy log while the apiserver part was running and eventually failed. It's not using the http(s)_proxy settings. Here's a couple snippets of the error:

apache-maven-3.3.9/bin/mvnyjp apache-maven-3.3.9/conf/ apache-maven-3.3.9/conf/logging/ apache-maven-3.3.9/conf/logging/simplelogger.properties apache-maven-3.3.9/conf/settings.xml apache-maven-3.3.9/conf/toolchains.xml apache-maven-3.3.9/lib/ext/ apache-maven-3.3.9/lib/ext/README.txt [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building db 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:09 min [INFO] Finished at: 2018-03-28T15:39:32+00:00 [INFO] Final Memory: 9M/111M [INFO] ------------------------------------------------------------------------ [ERROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5 from/to central (https://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:443 [repo.maven.apache.org/151.101.44.215] failed: Connection timed out -> [Help 1]

Mar 28, 2018 3:41:54 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host Mar 28, 2018 3:41:54 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {s}->https://repo.maven.apache.org:443

cdlliuy commented 6 years ago

@cwb124 , from the proxy logs, it looks like the dependencies are not downloaded correctly which led to the error of api packaging failure.

Could you check whether all the dependencies were downloaded correctly in your "blobs" folder? Here is an example of mine:

>> ls -l -R blobs
total 0
drwxr-xr-x  3 root  root   96 Sep 21  2017 golang
drwxr-xr-x  3 root  root   96 Sep 21  2017 maven
drwxr-xr-x  4 root  root  128 Sep 21  2017 nodejs
drwxr-xr-x  4 root  root  128 Sep 21  2017 openjdk

blobs/golang:
total 164112
-rw-------  1 root  root  84021919 Sep 21  2017 go1.7.4.linux-amd64.tar.gz

blobs/maven:
total 16592
-rw-------  1 root  root  8491533 Sep 21  2017 apache-maven-3.3.9-bin.tar.gz

blobs/nodejs:
total 41768
-rw-------  1 root  root  12041020 Sep 21  2017 node-v6.9.2-darwin-x64.tar.gz
-rw-------  1 root  root   9342556 Sep 21  2017 node-v6.9.2-linux-x64.tar.xz

blobs/openjdk:
total 299320
-rw-------  1 root  root  75550921 Sep 21  2017 openjdk-1.8.0_101-x86_64-mountainlion.tar.gz
-rw-------  1 root  root  77696717 Sep 21  2017 openjdk-1.8.0_101-x86_64-trusty.tar.gz
cwb124 commented 6 years ago

Blobs look good.

`ls -l -R blobs blobs: total 16 drwxrwxr-x 2 cbaum0001 staff 4096 Mar 28 15:27 golang drwxrwxr-x 2 cbaum0001 staff 4096 Mar 28 15:27 maven drwxrwxr-x 2 cbaum0001 staff 4096 Mar 28 15:27 nodejs drwxrwxr-x 2 cbaum0001 staff 4096 Mar 28 15:28 openjdk

blobs/golang: total 82056 -rw------- 1 cbaum0001 staff 84021919 Mar 28 15:27 go1.7.4.linux-amd64.tar.gz

blobs/maven: total 8296 -rw------- 1 cbaum0001 staff 8491533 Mar 28 15:27 apache-maven-3.3.9-bin.tar.gz

blobs/nodejs: total 20884 -rw------- 1 cbaum0001 staff 12041020 Mar 28 15:27 node-v6.9.2-darwin-x64.tar.gz -rw------- 1 cbaum0001 staff 9342556 Mar 28 15:27 node-v6.9.2-linux-x64.tar.xz

blobs/openjdk: total 149660 -rw------- 1 cbaum0001 staff 75550921 Mar 28 15:27 openjdk-1.8.0_101-x86_64-mountainlion.tar.gz -rw------- 1 cbaum0001 staff 77696717 Mar 28 15:28 openjdk-1.8.0_101-x86_64-trusty.tar.gz`

Here is a little more log detail around the failure. Again, it appears that the apiserver portion isn't honoring the proxy, or at least that's what it appears as I watch the squid proxy logs being active until it hits the apiserver portion, while traffic is leaving this linux box heading to maven.apache.org but without internet access, it dies.

`$ bosh create-release Blob download 'golang/go1.7.4.linux-amd64.tar.gz' (84 MB) (id: d51c7ac3-ee2b-4d9e-badb-b4474247fd18 sha1: 2e5baf03d1590e048c84d1d5b4b6f2540efaaea1) started Blob download 'golang/go1.7.4.linux-amd64.tar.gz' (id: d51c7ac3-ee2b-4d9e-badb-b4474247fd18) finished Blob download 'maven/apache-maven-3.3.9-bin.tar.gz' (8.5 MB) (id: 0c491801-2394-4610-bde5-b2334a84c52c sha1: 5b4c117854921b527ab6190615f9435da730ba05) started Blob download 'maven/apache-maven-3.3.9-bin.tar.gz' (id: 0c491801-2394-4610-bde5-b2334a84c52c) finished Blob download 'nodejs/node-v6.9.2-darwin-x64.tar.gz' (12 MB) (id: 837dfec9-2bb1-4c74-ab42-b4038d73e745 sha1: 6a8edc55c2e6246ad3ee1b80236bcdec94521115) started Blob download 'nodejs/node-v6.9.2-darwin-x64.tar.gz' (id: 837dfec9-2bb1-4c74-ab42-b4038d73e745) finished Blob download 'nodejs/node-v6.9.2-linux-x64.tar.xz' (9.3 MB) (id: 9cdab835-8968-4998-a0dd-b33065b03e47 sha1: 96b6e389d415f95b413dc7ffec7675546b738973) started Blob download 'nodejs/node-v6.9.2-linux-x64.tar.xz' (id: 9cdab835-8968-4998-a0dd-b33065b03e47) finished Blob download 'openjdk/openjdk-1.8.0_101-x86_64-mountainlion.tar.gz' (76 MB) (id: 76158715-a46b-4552-a6d4-ea8e9da9d2e0 sha1: d9374652844bfd7bcdce8f8b605fb57fd76c9e36) started Blob download 'openjdk/openjdk-1.8.0_101-x86_64-mountainlion.tar.gz' (id: 76158715-a46b-4552-a6d4-ea8e9da9d2e0) finished Blob download 'openjdk/openjdk-1.8.0_101-x86_64-trusty.tar.gz' (78 MB) (id: 6c3ab808-55e7-4555-aa59-2eb489b1200f sha1: d77ee059e289ae52fb87812f1a2932268427dc26) started Blob download 'openjdk/openjdk-1.8.0_101-x86_64-trusty.tar.gz' (id: 6c3ab808-55e7-4555-aa59-2eb489b1200f) finished Adding job 'pruner/a9b04121b5fab39485f4a681698b452da0118aee'... Added job 'pruner/a9b04121b5fab39485f4a681698b452da0118aee' Adding job 'scalingengine/4c4cfeeabd9e0c0e53b44a9563e209edf6eb6230'... Adding job 'eventgenerator/77dd4f5fece8bfd194b52f635cab5ca80ec01034'... Added job 'eventgenerator/77dd4f5fece8bfd194b52f635cab5ca80ec01034' Adding job 'metricscollector/85f0c39bcb9ef477c708d18802b4fbbfd097a7d5'... Adding job 'scheduler/eb373a92ec60baf1f63488eb3ec6cb9a4ef6141f'... Added job 'scalingengine/4c4cfeeabd9e0c0e53b44a9563e209edf6eb6230' Adding job 'apiserver/b05314ee056c084dc8cc5f1df532877b8468d62d'... Added job 'scheduler/eb373a92ec60baf1f63488eb3ec6cb9a4ef6141f' Added job 'apiserver/b05314ee056c084dc8cc5f1df532877b8468d62d' Added job 'metricscollector/85f0c39bcb9ef477c708d18802b4fbbfd097a7d5' Adding job 'servicebroker/3f2c7178367c0c54e69023d052cc7e0e3d5b4ca2'... Added job 'servicebroker/3f2c7178367c0c54e69023d052cc7e0e3d5b4ca2' Adding package 'common/306e7eb1a8187457885ce91eb4bc22f5aed734e8'... Added package 'common/306e7eb1a8187457885ce91eb4bc22f5aed734e8' Adding package 'eventgenerator/962b4333d1a427ea49837af7fa43148b027c502f'... Added package 'eventgenerator/962b4333d1a427ea49837af7fa43148b027c502f' Adding package 'metricscollector/8b86bbe2690bda5f7c5e20ff160fa2357c171e30'... Added package 'metricscollector/8b86bbe2690bda5f7c5e20ff160fa2357c171e30' Adding package 'nodejs/78d4ee5eeb7010fd7d15a1d2986992942940229f'... Added package 'nodejs/78d4ee5eeb7010fd7d15a1d2986992942940229f' Adding package 'pruner/ae873d993a688f191b9026637fa4b38abc64678f'... Added package 'pruner/ae873d993a688f191b9026637fa4b38abc64678f' Adding package 'scalingengine/ef68823d61fba545a8260214d9f743531d1675ba'... Added package 'scalingengine/ef68823d61fba545a8260214d9f743531d1675ba' Adding package 'golang1.7/651d77736c6087be1ca1df72eb8e4d2e701778f9'... Added package 'golang1.7/651d77736c6087be1ca1df72eb8e4d2e701778f9' Adding package 'java/d6f4a8bb4e3bfb6c4f3121f231f1a0c569d7fdaf'... Added package 'java/d6f4a8bb4e3bfb6c4f3121f231f1a0c569d7fdaf' Adding package 'servicebroker/8758549484bd195a0c0f326420cfc6faab09b37e'... Added package 'servicebroker/8758549484bd195a0c0f326420cfc6faab09b37e' Adding package 'apiserver/b0ab0e6e8317cd7292c8230d491e13f232f623c7'... Added package 'apiserver/b0ab0e6e8317cd7292c8230d491e13f232f623c7'

Building a release from directory '/u/cbaum0001/working/app-autoscaler-release':

qibobo commented 6 years ago

@cwb124 As your log showed that the error maven can not access its repo. Please set the following java environments on your machine and retry: export _JAVA_OPTIONS="-Dhttps.proxyHost=host -Dhttps.proxyPort=port"

cdlliuy commented 5 years ago

Is the problem still here? @cwb124

boyang9527 commented 4 years ago

It has been a while. Pls reopen if the issue still exists.