search

ministryofjustice / cloud-platform

Documentation on the MoJ cloud platform
MIT License
86 stars 44 forks source link

CCCD negative deploy result (but not failed) #174

Closed jsugarman closed 6 years ago

jsugarman commented 6 years ago

FAO: @lukaszraczylo

Service name

Claim for crown court defence

CCCD/ADP

Service environment

Impact on the service

no immediate problem but repeated attempts at deploy to production resulting in negative deploy (non-zero return) despite the docker image having been deployed and, apparently, running ok

Problem description

seeing a weird thing in the adp deploy log https://ci.service.dsd.io/view/ADP/job/advocate-defence-payments-deploy/4005/consoleFull

whereby one instance is having problems with “Cannot allocate memory”

I notice that some instances are running zero_downtime_deploy true, others false. The problematic one is true

[52.211.235.81] out:     zero_downtime_deploy:
[52.211.235.81] out:         True
[52.211.235.81] out:           ID: docker
[52.211.235.81] out:     Function: service.running
[52.211.235.81] out:       Result: False

Contact person

@jsugarman @colinbruce @lostie @cccd_development

colinbruce commented 6 years ago

Notes: the instance that failed also stopped reporting data to grafana at 8:36 this morning... https://grafana.service.dsd.io/dashboard/db/help-with-fees?orgId=1&var-project=adp-gamma&var-host=i-0924435c82bff91cc_gamma_advocatedefencepayments&var-disk=xvda1

lukaszraczylo commented 6 years ago

Contacted @jsugarman to discuss. Can't reproduce. Build 4007 worked fine.