cloudfoundry / cf-deployment

The canonical open source deployment manifest for Cloud Foundry
Apache License 2.0
294 stars 306 forks source link

connection reset by peer on CF v6.0 #920

Closed riddhichheda closed 3 years ago

riddhichheda commented 3 years ago

Hi Team, We are using v6.0 of cf-deployment stemcell bosh-aws-xen-hvm-ubuntu-xenial-go_agent/170.3 with cf routing release v0.182.0 bosh version 270.5.0 on AWS. We have kept the router in public subnet (For some testing purpose only otherwise it is as the backend of load-balancer). We have a third party identity management in-front of CF. We have SSL termination on router. And for some reason an internal component of CF is sending connection reset by peer messages to the identity management servers with the app URL. Is it possible that the router itself will be resetting the connection or rejecting the packets? Are there any such issues reported with this routing release or cf version? The issue seems to come on randomly. Random requests fails and there are no entry in router or app logs for the same. (Also connection reset by peer message on the identity management servers are coming in both the cases - with and without a public router).

Errpr message on Identity management servers: 2021-02-12T11:21:36,448 ERROR [CcRiTjRZiCeNFVGSDPdAyA] .transport.http.InternalHttpClient - Exchange to https:// failed: IOException: Connection reset by peer

Bosh Deployment command: bosh -e cf-env -d cf-deployment-test deploy cf-deployment/cf-deployment.yml -o cf-deployment/operations/aws.yml -o custom-ops/modify-network-and-ip.yml -o cf-deployment/operations/override-app-domains.yml -o cf-deployment/operations/set-bbs-active-key.yml -o cf-deployment/operations/community/change-metron-agent-deployment.yml -o custom-ops/modify-dns-aliases.yml -o cf-deployment/operations/use-external-dbs.yml -o cf-deployment/operations/use-external-blobstore.yml -o cf-deployment/operations/use-s3-blobstore.yml -o custom-ops/modify-manifest.yml -o custom-ops/modify-azs.yml -o custom-ops/add-vcap-password.yml -o custom-ops/enable-syslog-forwader.yml -o custom-ops/add-org-quota-defination.yml -o custom-ops/add-password-policy.yml -o custom-ops/remove-tcp-router.yml -o custom-ops/add-fog-connections.yml -o custom-ops/add-additional-routers.yml -o custom-ops/add-cf-exporter.yml -o cf-deployment/operations/community/change-metron-agent-deployment.yml -o custom-ops/modify-variables-valid-duration.yml -o custom-ops/add-custom-buildpack.yml -l vars_env.yml

This is our router properties section from manifest: router: backends: cert_chain: ((gorouter_backend_tls.certificate)) enable_tls: true private_key: ((gorouter_backend_tls.private_key)) ca_certs: | ((router_ca.certificate)) ((application_ca.certificate)) ((service_cf_internal_ca.certificate)) cipher_suites: client_cert_validation: require disable_http: true dummy_ca: | ((router_ca.certificate)) enable_ssl: true forwarded_client_cert: sanitize_set min_tls_version: TLSv1.1 route_services_recommend_https: false route_services_secret: ((router_route_services_secret)) status: password: ((router_status_password)) user: router-status tls_pem:

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177041650

The labels on this github issue will be updated when the story is started.

matt-royal commented 3 years ago

Hello @riddhichheda. cf-deployment v6.0 is over 2 years old, so none of us have recent experience with it. It's possible that your issue would be resolved by upgrading to a more modern version of cf-deployment.

You may be able to further debug this issue by looking at the logs for the go-router to see if there's any indication why the connection is being reset. Also feel free to reach out on the cloudfoundry slack for help. The #cf-deployment channel may be a good place to start.

davewalter commented 3 years ago

Hi @riddhichheda,

I am going to close this issue due to lack of activity. Please feel free to re-open it (or create a new one) if you are still having trouble after upgrading to a more recent version. Unfortunately, we are not able to support older versions of cf-deployment.

Regards, Dave