search

awslabs / aws-service-catalog-puppet

This is a framework where you list your AWS accounts with tags and your AWS Service Catalog products with tags or target accounts. The framework works through your lists, dedupes and spots collisions and then provisions the products into your AWS accounts for you. It handles the Portfolio sharing, its acceptance and can provision products cross account and cross region.
Apache License 2.0
75 stars 41 forks source link

Tooling account Puppet-deploy-in-spoke stuck running "portfolio-get-all-products-and-their-version" against all products. #664

Closed myshauno closed 1 year ago

myshauno commented 1 year ago

Describe the bug After updating to 0.231.0 from 0.171.0 we are experiencing an issue with the servicecatalog-puppet-deploy-in-spoke codebuild project running portfolio-get-all-products-and-their-version against each account for each individual product in each portfolio it seems.

This is only effecting the servicecatalog-puppet-deploy-in-spoke codebuild project within the tooling account.

Example of an entry below:

INFO MainProcess scheduler sending: portfolio-get-all-products-and-their-versions-ACCOUNTID-launches-vpc-flow-logs-MFI-cicd--eu-central-1-shared-guardrails
INFO MainProcess worker#2 executing task: portfolio-get-all-products-and-their-versions-ACCOUNTID-launches-vpc-flow-logs-rock-cicd--eu-central-1-shared-guardrails
INFO MainProcess worker#2 GetAllProductsAndTheirVersionsTask:portfolio-get-all-products-and-their-versions-ACCOUNTID-launches-vpc-flow-logs-rock-cicd--eu-central-1-shared-guardrails started
INFO MainProcess worker#2 executed task [success]: portfolio-get-all-products-and-their-versions-ACCOUNTID-launches-vpc-flow-logs-rock-cicd--eu-central-1-shared-guardrails got lock to unlock resources
INFO MainProcess scheduler receiving: [300]: portfolio-get-all-products-and-their-versions-ACCOUNTID-launches-vpc-flow-logs-rock-cicd--eu-central-1-shared-guardrails, COMPLETED

I assume this may have to do with the current structure of our puppet repo, as I've been resolving a few issues, we had to remove our spoke-local-portfolios folder just to get the run working initially.

To Reproduce Steps to reproduce the behavior:

  1. Conduct update on puppet-initialization-stack, setting the version parameter to: aws-service-catalog-puppet==0.231.0
  2. Await successful update notification in CloudFormation stack
  3. Go to CodePipeline to run servicecatalog-puppet-pipeline
  4. See error

Expected behavior Hoping for a successful run which doesn't hit the codebuild timeout.

Attachments Please let me know what files might assist with diagnosis, I can provide our manifest/insight into our code if/when required.

eamonnfaherty commented 1 year ago

Could you please share your manifest files and your full logs.

myshauno commented 1 year ago

Hey Eamonn,

I've sent you the manifest and logs to your AWS email with the subject "#664 aws-service-catalog-puppet issue" If this isn't suitable just let me know.

The Puppet-deploy logs cut off early, but repeats the last line until the run ended at 23:26