DeployHubProject / DeployHub-Pro

DeployHub Pro Pipeline Status Project
https://www.openmakesoftware.com/application-release-automation-for-continuous-delivery/
Other
11 stars 4 forks source link

deployments getting stuck #301

Closed piyush94 closed 4 years ago

piyush94 commented 4 years ago

Hi Steve, we have ran into a very weird issue. Components are not getting deployed after count 33. In /tmp i can see this image

no new ansible processes get created after this and log is stuck like this, everytime: image

I am also not able to stop the deployments, as i can't see any processes in the endpoint or deployhub container.

Kindly help with this on priority.

CC: @svisagan83

sbtaylor15 commented 4 years ago

@piyush94 is this only happening to one application or all applications?

CC: @svisagan83

piyush94 commented 4 years ago

@sbtaylor15 not tried with other applications. Currently this is only application of type Helm chart and with these many components.

CC: @svisagan83

sbtaylor15 commented 4 years ago

@piyush94 send us a screen shot of the component versions associated to the application version.

piyush94 commented 4 years ago

@sbtaylor15 image

piyush94 commented 4 years ago

@sbtaylor15 33 might be dropzone count, as i am seeing total 33 components

piyush94 commented 4 years ago

@sbtaylor15 the top two file and folder are getting created every second and getting deleted image

I think it would be good, if we can stop the earlier deployments first.

sbtaylor15 commented 4 years ago

@piyush94 did any of it deploy? if so how far did it get before getting stuck?

Those two are the Ansible files being created. The other directories are the drop zone locations.

piyush94 commented 4 years ago

@sbtaylor15 seeing helm list output, i think upto some point it's getting deployed. image

image

In logs cacheservice is the last deployed and it matches helm list output.

sbtaylor15 commented 4 years ago

@piyush94 - can we do a web meeting in 15 mins? 10:30 MST

piyush94 commented 4 years ago

@sbtaylor15 ok sure.

sbtaylor15 commented 4 years ago

@piyush94 send zoom link via email

sbtaylor15 commented 4 years ago

@piyush94 fyi, we were able to recreate the issue. It has to do with the number of threads. working on a resolution.

CC: @svisagan83

piyush94 commented 4 years ago

@sbtaylor15 Great, thanks!

CC: @svisagan83

sbtaylor15 commented 4 years ago

@piyush94 the thread limit is set in the database. Run this query which will set the limit to 256. Also, we are making a change to the C++ code to free up the threads. The new exe is NOT mandatory but the query update is.

update dm.dm_engineconfig set name = 'thread_limit', value = 256;

CC: @svisagan83

piyush94 commented 4 years ago

@sbtaylor15 - Thanks, deployment is working now.

CC: @svisagan83

sbtaylor15 commented 4 years ago

@piyush94 - what was the issue and how did you solve it?

piyush94 commented 4 years ago

@sbtaylor15 - In assigned components, connection line was missing between two of the components.