Scripts Sometimes fails. Second try works

[Toonix]

Sometimes it fails mostly second time it then works:

job_deploy: stage: deploy dependencies:

• job_build_image image: cdrx/rancher-gitlab-deploy script:
• upgrade --environment dod --stack bv-bva --service bv-bva-app

Skipping Git submodules setup $ upgrade --environment dod --stack bv-bva --service bv-bva-app Upgrading bv-bva/bv-bva-app in environment dod... Upgrade started, waiting for upgrade to complete... Traceback (most recent call last): File "/usr/local/bin/upgrade", line 11, in load_entry_point('rancher-gitlab-deploy==1.4', 'console_scripts', 'rancher-gitlab-deploy')() File "/usr/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke return callback(args, kwargs) File "/usr/local/lib/python2.7/site-packages/rancher_gitlab_deploy-1.4-py2.7.egg/rancher_gitlab_deploy/cli.py", line 212, in main api, environment_id, service['id'] File "/usr/local/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/api.py", line 72, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/api.py", line 58, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 508, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python2.7/site-packages/requests-2.18.4-py2.7.egg/requests/adapters.py", line 490, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer')) ERROR: Job failed: exit code 1

[cdrx]

Is there a load balancer or proxy of some sort in front of your Rancher instance?

[Toonix]

Yes, but not inside of gitlab, we use there the direct link RANCHER_URL=http://172.16.16.130:8080 so no...no proxy

[cdrx]

How much RAM does your Rancher instance have? I think it might not have enough resources

Normally I would suggest adding a longer timeout with --upgrade-timeout but because its Rancher hanging up the connection rather than rancher-gitlab-deploy I don't think that will help here

[Toonix]

Hmm 2GB of Ram 2 CPU for the rancher server And 8 GB for the Rancher Hosts

never had a issue connection with chrom to the rancher.

[cdrx]

Whats in your Rancher logs around the same time that the upgrade fails?

[Toonix]

Looks like we found it....it's Rancher 1.3.3 and 1.3.4 who dosn't run well...

Up 10 hours 3306/tcp, 0.0.0.0:8080->8080/tcp rancher-server Up 18 hours 3306/tcp, 0.0.0.0:8080->8080/tcp rancher-server

Maybe it's time to upgrade rancher on 1.6.7 dosn't have this restarts

Up 2 weeks 3306/tcp, 0.0.0.0:8080->8080/tcp rancher-server